Summation with Recursion

Let’s turn to some examples. To sum a list (or other sequence) of numbers, we can either use the built-in sum function or write a more custom version of our own. Example 19-1 shows what a custom summing function might look like when coded with recursion.

def mysum(L):
    if not L:
        return 0
    else:
        return L[0] + mysum(L[1:])       # Call myself recursively

To use, either add self-test code to the bottom of this file and run it as a script, or import it as a module and test at the REPL (again, the file may need to be in the folder where you’re working either way, per Chapter 3). With the latter:

>>> from mysum import mysum              # Import file as a module in a REPL
>>> mysum([1, 2, 3, 4, 5])               # Sum all the numbers in any sequence
15

At each level, this function calls itself recursively to compute the sum of the rest of the list, which is later added to the item at the front. This recursive loop ends and zero is returned when the list becomes empty. When using recursion like this, each open level of call to the function has its own copy of the function’s local scope on the runtime call stack. Here, that means L is different in each level, so each remembers its own segment of the list.

If this is difficult to understand (and it often is for new programmers), try adding a print of L to the function and run it again, to trace the current list at each call level; here’s the required mod pasted at the REPL for variety:

>>> def mysum(L):
        print(L)                         # Trace recursive levels
        if not L:                        # L shorter at each level
            return 0
        else:
            return L[0] + mysum(L[1:])

>>> mysum([1, 2, 3, 4, 5])
[1, 2, 3, 4, 5]
[2, 3, 4, 5]
[3, 4, 5]
[4, 5]
[5]
[]
15

As you can see, the list to be summed grows smaller at each recursive level, until it becomes empty—the termination of the recursive loop. The sum is then computed as the recursive calls unwind on returns.

Coding Alternatives

Interestingly, we can use Python’s if/else ternary expression (described in Chapter 12) to save some code real estate here. We can also generalize for any summable type (which is easier if we assume at least one item in the input) and use extended unpacking assignment to make the first/rest unpacking simpler (as covered in Chapter 11). Example 19-2 collects all three of these mods, ready to be run in a file or pasted into a REPL.

def mysum(L):
    return 0 if not L else L[0] + mysum(L[1:])           # Use ternary expression

def mysum(L):
    return L[0] if len(L) == 1 else L[0] + mysum(L[1:])  # Any type, assume one+

def mysum(L):
    first, *rest = L
    return first if not rest else first + mysum(rest)    # Use extended unpacking

When tested individually, all three of these alternatives handle numeric summation the same as the original:

>>> mysum([1, 2, 3, 4, 5])
15

Uniquely, the latter two fail for empties (e.g., mysum([])), but handle sequences of any object type that supports +, not just numbers (for strings, the effect is similar to ''.join(L)):

>>> mysum(('h', 'a', 'c', 'k'))          # The last two fail on mysum([])
'hack'
>>> mysum(['hack', 'app', 'code'])       # But they support nonnumeric types
'hackappcode'

Run some tests on your own for more insight. If you study these three variants, you’ll also find that:

• The latter two work on a single string argument (e.g., mysum('hack')), because strings are sequences of one-character strings (though this use case isn’t very useful: you get back the same string).

• The third variant also works on arbitrary iterables, including open input files (mysum(open(name))), but the others’ indexing generally fails on nonsequences (see Chapter 14 for extended unpacking demos).

You may also notice that the third variant’s unpacking assignment is similar to a * collector in a function header, and it’s tempting to recode it as such. This won’t quite work, though, because it would expect individual arguments, not a single iterable—unless we also star both the top-level input and recursive call. Here’s the end result, though by summing discrete arguments, it solves a different problem than both the prior versions and built-in sum:

>>> def mysum(first, *rest):
        return first if not rest else first + mysum(*rest)

>>> mysum(*[1, 2, 3, 4, 5])
15
>>> mysum(*'hack')
'hack'

Finally, bear in mind that recursion can be either direct, as in the examples so far, or indirect, as in the following—a function that calls another function, which calls back to its caller. The net effect is the same, though there are two function calls at each level instead of one:

>>> def mysum(L):
        if not L: return 0
        return nonempty(L)                  # Call a function that calls me

>>> def nonempty(L):
        return L[0] + mysum(L[1:])          # Indirectly recursive

>>> mysum([1.1, 2.2, 3.3, 4.4])
11.0

Loop Statements Versus Recursion

Though recursion works for summing in the prior sections’ examples, it’s probably overkill in this context. In fact, recursion is not used nearly as often in Python as in more esoteric languages like Prolog or Lisp, because Python emphasizes simpler procedural statements like loops, which are usually more natural. The while, for example, often makes things more concrete, and it doesn’t require that a function be defined to allow recursive calls:

>>> L = [1, 2, 3, 4, 5]
>>> tot = 0
>>> while L:
        tot += L[0]
        L = L[1:]

>>> tot
15

Better yet, for loops iterate for us automatically, making recursion largely extraneous in many cases (and, in all likelihood, less efficient in terms of memory space and execution time):

>>> L = [1, 2, 3, 4, 5]
>>> tot = 0
>>> for x in L: tot += x

>>> tot
15

With looping statements, we don’t require a fresh copy of a local scope on the call stack for each iteration, and we avoid the speed costs associated with function calls in general. (Stay tuned for Chapter 21’s timer case study for ways to compare the execution times of alternatives like these.)

Handling Arbitrary Structures

On the other hand, recursion—or equivalent and explicit stack-based algorithms we’ll explore shortly—can be required to traverse arbitrarily shaped structures. As a simple example of recursion’s role in this context, consider the task of computing the sum of all the numbers in a nested sublists structure like this:

[1, [2, [3, 4], 5], 6, [7, 8]]                  # Arbitrarily nested sublists

Neither our prior summers nor simple looping statements will work here because this is not a linear iteration. Nested looping statements do not suffice either—because the sublists may be nested to arbitrary depth and in an arbitrary shape, there’s no way to know how many nested loops to code to handle all cases. Instead, the function in Example 19-3 accommodates such general nesting by using recursion to visit sublists along the way.

def sumtree(L, trace=False):
    tot = 0
    for x in L:                                 # For each item at this level
        if not isinstance(x, list):
            tot += x                            # Add numbers directly
            if trace: print(x, end=', ')
        else:
            tot += sumtree(x, trace)            # Recur for sublists
    return tot

In this file’s function, each recursive level runs a for loop to add numbers, or recur into sublists to open new levels. Recall from Chapter 9 that isinstance compares object types; it’s used here to detect nested sublists.

This code is also instrumented to trace items as they are added to the total: if trace is passed a true value, you can see how the object is scanned left to right. Because it also steps down into sublists with recursion along the way, though, the traversal is really both horizontal and vertical:

>>> from sumtree import sumtree
>>> sumtree([1, [2, [3, 4], 5], 6, [7, 8]])
36
>>> sumtree([1, [2, [3, 4], 5], 6, [7, 8]], trace=True)
1, 2, 3, 4, 5, 6, 7, 8, 36

tests = (
[1, [2, [3, 4], 5], 6, [7, 8]],      # Mixed nesting => 36
[1, [2, [3, [4, [5]]]]],             # Right-heavy nesting => 15
[[[[[1], 2], 3], 4], 5])             # Left-heavy nesting => 15

def tester(sumtree, trace=True):
    for test in tests:
        print(sumtree(test, trace))

>>> from sumtree import sumtree              # Get the summer
>>> from sumtree_tester import tester        # Get the tester
>>> tester(sumtree)                          # Run the tester on the summer
1, 2, 3, 4, 5, 6, 7, 8, 36
1, 2, 3, 4, 5, 15
1, 2, 3, 4, 5, 15

def sumtree(L, trace=False):                     # Breadth-first, explicit queue
    tot = 0
    items = list(L)                              # Start with copy of top level
    while items:
        front = items.pop(0)                     # Fetch/delete front item
        if not isinstance(front, list):
            tot += front                         # Add numbers directly
            if trace: print(front, end=', ')
        else:
            items.extend(front)                  # <== Append all in nested list
    return tot

>>> from sumtree_queue import sumtree            # Get the new summer
>>> sumtree([1, [2, [3, 4], 5], 6, [7, 8]])
36
>>> from sumtree_tester import tester            # Unless already imported 
>>> tester(sumtree)                              # Run tester on _this_ summer
1, 6, 2, 5, 7, 8, 3, 4, 36
1, 2, 3, 4, 5, 15
5, 4, 3, 2, 1, 15

def sumtree(L, trace=False):                     # Depth-first, explicit stack
    tot = 0
    items = list(L)                              # Start with copy of top level
    while items:
        front = items.pop(0)                     # Fetch/delete front item
        if not isinstance(front, list):
            tot += front                         # Add numbers directly
            if trace: print(front, end=', ')
        else:
            items[:0] = front                    # <== Prepend all in nested list
    return tot

>>> from sumtree_stack import sumtree            # Same name, different file
>>> sumtree([1, [2, [3, 4], 5], 6, [7, 8]])
36
>>> from sumtree_tester import tester            # Optional if already imported
>>> tester(sumtree)
1, 2, 3, 4, 5, 6, 7, 8, 36
1, 2, 3, 4, 5, 15
1, 2, 3, 4, 5, 15

>>> L = [1, 2]
>>> L.append(L)      # Make a cyclic object: L references itself
>>> L
[1, 2, [...]]

>>> from sumtree import sumtree
>>> sumtree(L)
RecursionError: maximum recursion depth exceeded

>>> from sumtree_queue import sumtree
>>> sumtree(L)
…hang or crash, and ditto for stack…

  if state not in visited:
      visited.add(state)          # x.add(state), x[state]=True, or x.append(state)
      …proceed…

  visited.add(front)
  …proceed…
  items.extend([x for x in front if x not in visited])

>>> sys.getrecursionlimit()         # 1000 calls deep default
1000
>>> sys.setrecursionlimit(10000)    # Allow deeper nesting
>>> help(sys.setrecursionlimit)     # Read more about it

The First-Class Object Model

Because Python functions are objects, you can write programs that process them generically. Function objects may be reassigned to other names, passed to other functions, embedded in data structures, returned from one function to another, and more, as if they were simple numbers or strings. Function objects happen to support a special operation—they can be called by listing arguments in parentheses—but they belong to the same general category as other objects.

As we’ve seen, this is usually called a first-class object model; it’s ubiquitous in Python, and a necessary part of functional programming. We’ll explore this programming mode more fully in this and the next chapter; because its motif is founded on the notion of applying functions, it treats functions as a kind of data.

We’ve explored some generic use cases for functions in earlier examples, but a quick review helps to underscore the model. For example, there’s really nothing special about the name used in a def statement: it’s just a variable assigned in the current scope, as if it had appeared on the left of an = sign. Because the function name is simply a reference to an object after a def runs, you can reassign that object to other names freely and call it through any reference:

>>> def exclaim(message):              # Name exclaim assigned to function object
        print(message + '!')

>>> exclaim('Direct call')             # Call object through original name
Direct call!

>>> x = exclaim                        # Now x references the function too
>>> x('Indirect call')                 # Call object through name x by adding ()
Indirect call!

And because arguments are passed by assigning objects, it’s just as easy to pass functions to other functions as arguments. The callee may then call the passed-in function just by adding arguments in parentheses (see the earlier summer tester in Example 19-4 for another example of this pattern):

>>> def generic(func, arg):
        func(arg)                      # Call the passed-in object by adding ()

>>> generic(exclaim, 'Argument call')  # Pass the function to another function
Argument call!

You can even stuff function objects into data structures, as though they were integers or strings. The following, for example, embeds the function twice in a list of tuples, as a sort of actions table. Because Python compound types like these can contain any sort of object, there’s no special case here, either (Example 18-2 used similar code):

>>> schedule = [ (exclaim, 'Hack'), (exclaim, 'Code') ]
>>> for (func, arg) in schedule:
        func(arg)                      # Call functions embedded in containers

Hack!
Code!

This code simply steps through the schedule list, calling the exclaim function with one argument each time through. As we saw in Chapter 17’s examples, functions can also be created and returned for use elsewhere—the closure created in this mode also retains state from the enclosing scope:

>>> def implore(verb):                 # Make a function but don't call it
        def exclaim(noun):
            print(f'{verb} {noun}!')
        return exclaim

>>> I = implore('Hack')                # Label in enclosing scope is retained
>>> I('Code')                          # Call the function that make returned
Hack Code!
>>> I('App')
Hack App!

Python’s first-class object model and lack of type constraints make for a very flexible programming language.

Function Introspection

In fact, functions are more flexible than you might expect. Because they are objects, we can also process functions with normal object tools. For instance, by now we know that once we make a function we can call it as usual:

>>> def func(a):
        b = 'Hack'
        return b * a

>>> func(8)
'HackHackHackHackHackHackHackHack'

But the call expression is just one of a set of operations defined to work on function objects. For instance, we can also inspect their attributes generically:

>>> func.__name__
'func'
>>> dir(func)
'__annotations__', '__builtins__', '__call__', '__class__', '__closure__',
…more omitted: 38 total…
'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__type_params__']

Introspection—internals access—tools like this allow us to explore implementation details. For example, functions have attached code objects, which provide details on aspects such as the functions’ local variables and arguments:

>>> func.__code__
<code object func at 0x103ef3910, file "<stdin>", line 1>

>>> dir(func.__code__)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', 
…more omitted: 48 total…
'co_posonlyargcount', 'co_qualname', 'co_stacksize', 'co_varnames', 'replace']

>>> func.__code__.co_varnames
('a', 'b')
>>> func.__code__.co_argcount
1

Tool writers can make use of such information to manage functions—in fact, we will too in the online-only “Decorators” chapter, to implement validation of function arguments with the decorators introduced ahead. Whether you code or use such tools, object introspection boosts function utility.

Function Attributes

Nor are function objects limited to the system-defined attributes of the prior section: it’s also possible to attach arbitrary user-defined attributes to them. This topic was introduced in Chapter 17, but this section expands on it with additional context and examples. As usual in Python, function attributes are created by simple assignments:

>>> def func(): pass
>>> func
<function func at 0x1043771a0> 
>>> func.count = 0
>>> func.count += 1
>>> func.count
1
>>> func.handles = 'Button-Press'
>>> func.handles
'Button-Press'
>>> dir(func)
…most double-underscore names omitted…
'__str__', '__subclasshook__', '__type_params__', 'count', 'handles']

Python’s own implementation-related data stored on functions follows naming conventions that prevent them from clashing with the more arbitrary attribute names you might assign yourself. Specifically, all function internals’ names have leading and trailing double underscores (“__X__”):

>>> len(dir(func))
40
>>> [x for x in dir(func) if not x.startswith('__')]
['count', 'handles']

If you’re careful not to name attributes the same way as Python, you can safely use the function’s namespace as though it were your own namespace or scope. Naturally, all of this works the same for functions made with lambda:

>>> F = lambda: None
>>> len(dir(F))
38
>>> F.book = 'LP6E'
>>> F.book
'LP6E'

As covered in Chapter 17, such attributes can be used to attach state information to function objects directly, instead of using other techniques such as globals, nonlocals, and classes. Unlike nonlocals, such attributes are accessible anywhere the function itself is, even from outside its code.

In a sense, this is also a way to emulate “static locals” in other languages—variables whose names are local to a function, but whose values are retained after a function exits. Attributes are related to objects instead of scopes (and must be referenced through the function name within its code), but the net effect is similar.

Moreover, as also explored in Chapter 17, when attributes are attached to functions generated by other factory functions, they also support multiple copy, writeable, and per-call state retention, as an alternative to closures and class-instance attributes. This makes function attributes a broadly useful tool—like the topics of the next section.

Function Annotations and Decorations

For tools roles, functions also support attached annotations—arbitrary user-defined info about a function’s arguments and result that augment the function. Python provides syntax for coding annotations, but it doesn’t do anything with them itself—annotations are completely optional, don’t impact function behavior in any way, and when present are simply attached to the function object’s __annotations__ attribute for use by other tools.

While not of general interest to most application programmers, third-party tools and libraries might use annotations in the context of enhanced error checking, or API directives. Type hinting, discussed in Chapter 6, is also based on annotations, but is optional and unused, and doesn’t preclude other roles for annotations today (more on this ahead).

We studied the formal rules for arguments in function definitions in the preceding chapter. Annotations don’t modify these rules but simply extend their syntax to allow extra expressions to be associated with named arguments and function results. Consider the following nonannotated function, coded with three arguments and a returned result:

>>> def func(a, b, c):
        return a + b + c

>>> func(1, 2, 3)
6

Syntactically, function annotations are coded in def header lines (only), as arbitrary expressions associated with arguments and return values. For arguments, they appear after a colon immediately following the argument’s name; for return values, they are written after a -> following the arguments list’s closing parenthesis. The following code, for example, annotates all three of the prior function’s arguments, as well as its return value:

>>> def func(a: 'hack', b: (1, 10), c: float) -> int:
        return a + b + c

>>> func(1, 2, 3)
6

Calls to an annotated function work as usual, but when annotations are present Python collects them in a dictionary and attaches it to the function object itself as its __annotations__. In this, argument names become keys; the return value annotation is stored under key return if coded (which suffices because this reserved word can’t be used as an argument name); and the values of annotation keys are assigned to the results of the annotation expressions:

>>> func.__annotations__
{'a': 'hack', 'b': (1, 10), 'c': <class 'float'>, 'return': <class 'int'>}

Because they are just Python objects attached to a Python object, annotations are straightforward to process. The following annotates just two of three arguments and steps through the attached annotations generically:

>>> def func(a: 'python', b, c: 3.12):
        return a + b + c

>>> func(1, 2, 3)
6
>>> func.__annotations__
{'a': 'python', 'c': 3.12}

>>> for arg in func.__annotations__:
       print(arg, '=>', func.__annotations__[arg])

a => python
c => 3.12

There are two fine points to note here. First, you can still use defaults for arguments if you code annotations—the annotation (and its : character) appear before the default (and its = character). In the following, for example, a: 'hack' = 4 means that argument a defaults to 4 and is annotated with the string 'hack':

>>> def func(a: 'hack' = 4, b: (1, 10) = 5, c: float = 6) -> int:
        return a + b + c

>>> func(1, 2, 3)                # No defaults used
6
>>> func()                       # a=4 + b=5 + c=6 (all defaults)
15
>>> func(1, c=10)                # 1 + b=5 + 10 (keywords work normally)
16
>>> func.__annotations__
{'a': 'hack', 'b': (1, 10), 'c': <class 'float'>, 'return': <class 'int'>}

Second, note that the blank spaces in the prior example are all optional—you can use spaces between components in function headers or not, but omitting them might degrade your code’s readability to some observers (and probably improve it to others):

>>> def func(a:'hack'=4, b:(1,10)=5, c:float=6)->int:
        return a + b + c

>>> func(1, 2)                   # 1 + 2 + c=6
9
>>> func.__annotations__
{'a': 'hack', 'b': (1, 10), 'c': <class 'float'>, 'return': <class 'int'>}

While annotations are optional and uncommon, they may be used to specify constraints for types or values, and larger APIs might use this feature as a way to register function interface information. In fact, you’ll see a potential application in the online-only “Decorators” chapter, when we code annotations as an alternative to function decorator arguments—a more general concept in which augmentation info is coded outside the function header and so is not limited to a single role.

@decorator
def func(args): …           # Decorated function def

def func(args): …
func = decorator(func)      # Rebind func to decorator's result

def echo(F):
    def proxy(*args): 
        print('calling', F.__name__)       # Add actions here
        return F(*args)                    # Run decorated function
    return proxy

@echo
def func(x, y):                            # Rebinds func = echo(func)
    print('I am running...', x, y)         # func is run by the proxy closure

>>> func(1, 2)                             # Runs a proxy, which runs func
calling func
I am running... 1 2

lambda Basics

As a refresher, the lambda’s general form is the keyword lambda, followed by zero or more arguments (just like the arguments you enclose in parentheses in a def header, sans annotations), followed by an expression after a colon:

lambda argument1, argument2,… argumentN : expression-using-arguments

Parentheses are not allowed around lambda arguments and are generally optional around the entire lambda itself, though they’re required in some contexts and may boost clarity in others. Functions returned by lambda work the same as those assigned by def, but lambda differs in ways that make it useful in specialized roles:

• lambda is an expression, not a statement. Because of this, a lambda can appear in places a def is not allowed by Python’s syntax—inside a list literal or a function call’s arguments, for example. With def, functions can be referenced by name in such locations, but must be created elsewhere. As an expression, lambda returns a value (a new function) that can be assigned a name, or used where the lambda appears.

• lambda’s body is a single expression, not a block of statements. The lambda’s body is similar to what you’d put in a def body’s return statement; you simply type the result as a naked expression, instead of explicitly returning it. Because it is limited to an expression, a lambda is less general than a def—you can only squeeze so much logic into a lambda body without full-blown statements. This is by design, to limit program nesting: lambda is designed for coding simple functions, and def handles larger tasks.

Apart from those distinctions, defs and lambdas do the same sort of work. For instance, by this point we should be pros at making a function with a def statement:

>>> def func(x, y, z): return x + y + z
>>> func(2, 3, 4)
9

But we can achieve the same effect with a lambda expression by explicitly assigning its result to a name through which you can later call the otherwise-anonymous function:

>>> func = lambda x, y, z: x + y + z
>>> func(2, 3, 4)
9

Here, func is manually assigned the function object the lambda expression creates; this is how def works, too, but its assignment is automatic. Defaults and other argument-matching syntax work in lambda too, just like in a def:

>>> x = (lambda a='hack', b='python', c='code': a + b + c)
>>> x('write')
'writepythoncode'

The code in a lambda body also follows the same scope lookup rules as code inside a def. lambda expressions introduce a new local scope much like a nested def, which automatically sees names in enclosing functions, the module, and the built-in scope—via the LEGB rule we studied in Chapter 17:

>>> def editions(title):                          # title in enclosing scope
        return (lambda e: title + ', ' + e)       # Return a function object

>>> labeler = editions('Learning Python')         # Make+save nested function
>>> labeler('6E')                                 # '6E' passed to e in lambda
'Learning Python, 6E'

With those basic lambda “hows” in hand, the natural next question is “why,” taken up in the next section.

Why Use lambda?

Generally speaking, lambda is a sort of function shorthand that allows you to embed a function’s definition within the code that uses it. It is entirely optional—you can always use def instead, and should if your function requires the power of full statements that the lambda’s expression cannot provide. But lambda may be easier to use in scenarios where you just need to embed a small bit of executable code inline at the place where it is to be later used.

For instance, you’ll see later that callback handlers are frequently coded as inline lambda expressions embedded directly in a registration call’s arguments list, instead of being defined with a def elsewhere in a file and referenced by name (see the sidebar “Why You Will Care: lambda Callbacks” for an example).

lambda is also commonly used to code jump tables, which are lists or dictionaries of actions to be performed on demand. For example:

L = [lambda x: x * 2,                # Inline function definition
     lambda x: x ** 2,
     lambda x: x // 2]               # A list of three callable functions

for f in L:
    print(f(5))                      # Prints 10, 25, and 2

print(L[1](5))                       # Prints 25

The lambda expression works well when you need to stuff small pieces of executable code into places where statements like def are not allowed. The preceding code snippet, for example, builds up a list of three functions by embedding lambda expressions inside a list literal; a def won’t work inside a literal like this because it is a statement, not an expression. The equivalent def coding would require temporary function names (which might clash with others) and function definitions outside the context of intended use (which might be hundreds of lines away):

def f1(x): return x * 2
def f2(x): return x ** 2             # Define named functions
def f3(x): return x // 4             # Separate from their place of use

L = [f1, f2, f3]                     # Reference by name

for f in L:
    print(f(2))                      # Also prints 10, 25, and 2

>>> key = 'loop'
>>> {'hack': lambda s: s.upper(),
     'code': lambda s: s.lower(),
     'loop': lambda s: f'{s * 4}!'}[key]('Py')
'PyPyPyPy!'

>>> def f1(s): return s.upper()
>>> def f2(s): return s.lower()
>>> def f3(s): return f'{s * 4}!'

>>> key = 'loop'
>>> {'hack': f1, 'code': f2, 'loop': f3}[key]('Py')
'PyPyPyPy!'

How (Not) to Obfuscate Your Python Code

The fact that the body of a lambda has to be a single expression (not a series of statements) would seem to place severe limits on how much logic you can pack into a lambda. If you know what you’re doing, though, you can code most statements in Python as expression-based equivalents.

For example, if you want to print from the body of a lambda function, simply use print or sys.stdout.write (recall from Chapter 11 that this is what print really does). And to execute multiple actions, code a sequence like a tuple, to evaluate nested multiple expressions from left to right (tuples require parentheses in this context):

>>> series = lambda a, b: (print(a.upper()), print(b.lower()))     # > 1 actions
>>> ignore = series('Hack', 'Code')
HACK
code

Similarly, to nest logic in a lambda, you can use the if/else ternary expression introduced in Chapter 12, or the equivalent but trickier and/or combination also described there. As you learned earlier, the following statement:

if a:
    b
else:
    c

can be emulated by either of these equivalent expressions (the second is only roughly the same, but close enough):

b if a else c
((a and b) or c)

Because expressions like these can be placed inside a lambda, they may be used to implement selection logic within a lambda function (the lambda is parenthesized here only for variety and subjective clarity):

>>> lower = (lambda x, y: x if x < y else y)         # Ifs in lambda: ternary
>>> lower('bb', 'aa')
'aa'
>>> lower('aa', 'bb')
'aa'

Furthermore, if you need to perform loops within a lambda, you can also embed things like list comprehension expressions and map calls—tools we met in earlier chapters and will revisit in this and the next chapter:

>>> showall = lambda x: [print(y) for y in x]        # Loops in lambda: list comp
>>> t = showall(['lp5e', 'lp6e'])
lp5e
lp6e
>>> showall = lambda x: list(map(print, x))          # Loops in lambda: maps
>>> t = showall(('py3.3', 'py3.12')) 
py3.3
py3.12
>>> showall = lambda x: print(*x, sep='', end='')    # Another option: *unpacking

And while it’s limited by the local scope of the function that lambda makes, assignment is in scope (pun intended) for lambda expressions that use the := named-assignment expression:

>>> namer = lambda x: (res := x + 1) + res           # Assignment in lambda: :=
>>> namer(2)
6
>>> res                                              # But it doesn't persist
NameError: name 'res' is not defined

There is a limit to emulating statements with expressions: you can’t assign nonlocals, for instance, though tools like the setattr built-in, the __dict__ of namespaces, and methods that change mutable objects in place can sometimes stand in—and can quickly lead you deep into the heart of expression-convolution darkness.

But now that this book has shown you these tricks, it must also humbly implore you to use them only as a last resort. Without due care, they can yield unreadable (a.k.a. obfuscated) Python code, despite the language’s best intents. In general, simple is better than complex, explicit is better than implicit, and full statements are better than arcane expressions. That’s why lambda is limited to expressions. If you have larger logic to code, use def; lambda is for small pieces of inline code. On the other hand, you may find these techniques useful—when taken in moderation.

Scopes: lambdas Can Be Nested Too

One last lambda note: as we saw in Chapter 17, lambda is also the main beneficiary of enclosing-function scope lookup—the E in the LEGB scope rule. As a review, in the following the lambda appears inside a def, its typical coding, and so can access the value that name x had in the enclosing function’s scope during its call:

>>> def action(x):
        return (lambda y: x + y)               # Make function, remember x

>>> act = action(99)
>>> act(2)                                     # Call what action returned
101

What wasn’t illustrated in the prior discussion of nested function scopes is that a lambda also has access to the names in any enclosing lambda. This case is somewhat obscure, but imagine if we recoded the prior def with a lambda:

>>> action = (lambda x: (lambda y: x + y))     # lambda scopes nest too
>>> act = action(99)
>>> act(3)
102
>>> ((lambda x: (lambda y: x + y))(99))(4)     # Even if you don't save them
103

Here, the nested lambda structure makes a function that makes a function when called. In both cases, the nested lambda’s code has access to the variable x in the enclosing lambda. This works, but it seems fairly convoluted code; in the interest of readability, nested lambdas may be best avoided. The good news, perhaps, is that def cannot be nested in lambda: because lambda’s body is an expression, statements like def won’t work—thankfully!

Mapping Functions over Iterables: map

One of the more common things programs do with lists and other sequences is apply an operation to each item and collect the results—selecting columns in database tables, incrementing pay fields of employees in a company, parsing email attachments, and so on. Python has multiple tools that make such collection-wide operations easy to code. For instance, we’ve seen that updating all the counters in a list can be done easily with a for loop:

>>> counters = [1, 2, 3, 4]

>>> updated = []
>>> for x in counters:
        updated.append(x + 10)                 # Add 10 to each item

>>> updated, counters
([11, 12, 13, 14], [1, 2, 3, 4])

But because this is such a common operation, Python also provides built-ins that do most of the work for you. Among them, the map function applies a passed-in function to each item in an iterable object and returns a list containing all the function-call results. For example, assuming counters in intact:

>>> def inc(x): return x + 10                  # Function to be run

>>> list(map(inc, counters))                   # Collect call results
[11, 12, 13, 14]

We met map briefly in Chapters 13 and 14, as a way to apply a built-in function to items in an iterable. Here, we make more general use of it by passing in a user-defined function to be applied to each item in the list—map calls our inc on each list item and collects all the return values into a new list. Remember that map is a nonsequence iterable, so a list call is used to force it to produce all its results for display here (per Chapter 14 coverage).

Because map expects a function to be passed in and applied, it also happens to be one of the places where lambda commonly appears:

>>> list(map((lambda x: x + 3), counters))     # Function expression
[4, 5, 6, 7]

Here, the function adds 3 to each item in the counters list; as this little function isn’t needed elsewhere, it was written inline as a lambda. Because such uses of map are equivalent to for loops, with a little extra code you can always code a general mapping utility yourself:

>>> def mymap(func, iter):
        res = []
        for x in iter: res.append(func(x))
        return res

Assuming the function inc is still as it was when it was shown previously, we can map it across a sequence (or other iterable) with either the built-in or our equivalent:

>>> list(map(inc, [1, 2, 3]))               # Built-in is an iterable
[11, 12, 13]
>>> mymap(inc, [1, 2, 3])                   # Ours builds a list (see generators)
[11, 12, 13]

However, as map is a built-in, it’s always available, always works the same way, and may have performance benefits (as we’ll prove in Chapter 21, it’s faster than a manually coded for loop in some usage modes). Moreover, map can be used in more advanced ways than shown here. For instance, given multiple sequence arguments, it sends items taken from sequences in parallel as distinct arguments to the function:

>>> pow(3, 4)                               # 3**4
81
>>> list(map(pow, [1, 2, 3], [2, 3, 4]))    # 1**2, 2**3, 3**4
[1, 8, 81]

With multiple sequences, map expects an N-argument function for N sequences. Here, the pow function takes two arguments on each call—one from each sequence passed to map. It’s not much extra work to simulate this multiple-sequence generality in code, too, but we’ll postpone doing so until later in the next chapter, after we’ve explored some additional iteration tools (hint: it would be better to generate items on demand, like the built-in).

The map call is also similar to the list comprehension expressions we studied in Chapter 14 and will revisit in the next chapter from a functional perspective:

>>> list(map(inc, [1, 2, 3, 4]))
[11, 12, 13, 14]
>>> [inc(x) for x in [1, 2, 3, 4]] 
[11, 12, 13, 14]

In some cases, map may be faster to run than a list comprehension, and it may also require less code. On the other hand, because map applies a function call to each item instead of an arbitrary expression, it is a somewhat less general tool, and often requires extra helper functions or lambdas. Moreover, wrapping a comprehension in parentheses instead of square brackets creates an object that generates values on request to save memory and increase responsiveness, much like map—a topic we’ll take up in the next chapter.

Selecting Items in Iterables: filter

The map function is a primary and relatively straightforward representative of Python’s functional programming toolset. Its close relatives, filter and reduce, select an iterable’s items based on a test function and apply functions to item pairs, respectively.

Because it also returns an iterable, filter (like range) requires a list call to display all its results in a REPL. For example, the following filter call picks out items in a sequence that are greater than zero:

>>> list(range(−5, 5)) 
[−5, −4, −3, −2, −1, 0, 1, 2, 3, 4]

>>> list(filter((lambda x: x > 0), range(−5, 5)))
[1, 2, 3, 4]

We met filter briefly earlier in the Chapter 12 sidebar “Why You Will Care: Booleans” and while exploring iterables in Chapter 14. Items in the sequence or iterable for which the function returns a true result are added to the result list. Like map, this function is roughly equivalent to a for loop, but it is built-in, concise, and often fast:

>>> res = []
>>> for x in range(−5, 5):                   # The statement equivalent of filter
        if x > 0:                            # Simple but slower today, probably
            res.append(x)

>>> res
[1, 2, 3, 4]

Also like map, filter can be emulated by list comprehension syntax with often simpler results (especially when it can avoid creating a new function), and with a similar generator expression when coded with enclosing parentheses to delay production of results—though again, the generator story will have to remain a teaser for the next chapter:

>>> [x for x in range(−5, 5) if x > 0]
[1, 2, 3, 4]

Combining Items in Iterables: reduce

The functional reduce call—once a built-in but now a resident of the functools standard library module—is more complex. It accepts an iterable to process, but it’s not an iterable itself: it returns a single result that aggregates an iterable’s items. To demo, here are two reduce calls that compute the sum and product of the items in a list:

>>> from functools import reduce 
>>> reduce((lambda x, y: x + y), [1, 2, 3, 4])
10
>>> reduce((lambda x, y: x * y), [1, 2, 3, 4])
24

At each step, reduce passes the current sum or product, along with the next item from the list, to the passed-in lambda function. By default, the first item in the sequence initializes the starting value. To make that more concrete again, here’s the for loop equivalent to the first of these calls, with the addition hardcoded inside the loop:

>>> L = [1, 2, 3, 4]
>>> res = L[0]
>>> for x in L[1:]:
        res += x

>>> res
10

In fact, coding your own reusable and customizable version of reduce is fairly straightforward too. The following function emulates most of the built-in’s behavior and helps demystify its operation in general:

>>> def myreduce(function, sequence):
        tally = sequence[0]
        for next in sequence[1:]:
            tally = function(tally, next)
        return tally

>>> myreduce((lambda x, y: x + y), [1, 2, 3, 4])
10
>>> myreduce((lambda x, y: x * y), [1, 2, 3, 4])
24

The built-in reduce also allows an optional third argument, effectively placed before the items in the sequence to serve as an initial value and a default result when the sequence is empty, but we’ll leave this extension as a suggested exercise (again, emulating built-in tools is instructive, but superfluous).

If this coding technique has sparked your interest, you might also be interested in the standard library operator module, which provides functions that correspond to built-in expressions and so comes in handy for some uses of functional tools (consult help in a REPL or Python’s library manual for more details on this module):

>>> import operator, functools
>>> functools.reduce(operator.add, [2, 4, 6])        # Function-based +
12
>>> functools.reduce((lambda x, y: x + y), [2, 4, 6])
12

Together, map, filter, and reduce support powerful functional programming techniques. As mentioned, many observers would also extend the functional programming toolset in Python to include the nested function closures and anonymous function lambdas we’ve explored, as well as the generators and comprehensions we’ve met in piecemeal fashion along the way. To fully grok the latter two, though, we must move on to the next chapter.