When not to use list comprehensions: Code obfuscation

With such generality, list comprehensions can quickly become, well, incomprehensible, especially when nested. Some programming tasks are inherently complex, and we can’t sugarcoat them to make them any simpler than they are (see the upcoming permutations for a prime example). Tools like comprehensions are powerful solutions when used wisely, and there’s nothing inherently wrong with using them in your scripts.

At the same time, code like that at the end of the prior section may introduce more complexity than it should—and, frankly, tends to disproportionately spark the interest of those holding the darker and misinformed assumption that code obfuscation somehow implies talent. In the interest of software citizenship, some perspective seems in order here.

This book demonstrates advanced comprehensions to teach, but in the real world, programming is not about being clever and obscure—it’s about how clearly your program communicates its purpose. Writing tricky comprehensions may be a fun academic recreation, but it doesn’t work in programs that others will someday need to understand.

In other words, the age-old acronym KISS applies here as always: Keep It Simple—traditionally followed either by a word that is now too sexist, or another that is too colorful for a G-rated book like this. If you have to translate code to simpler statements to understand it, it should probably be simpler statements in the first place.

When to use list comprehensions: Speed, conciseness, etc.

Nevertheless, in this case, there is currently a performance advantage to the extra complexity: based on tests run under Python today, map calls and list comprehensions can be significantly faster than equivalent for loops. This speed difference can vary per usage and Python, but is due to the fact that map and list comprehensions run at compiled-language speed inside the interpreter, which is generally faster than running for loop bytecode within the PVM.

Also in the plus column, list comprehensions offer a code conciseness that’s compelling and even warranted when that reduction in size doesn’t also imply a reduction in meaning for the next programmer; many find the expressiveness of comprehensions to be a powerful ally; and because map and list comprehensions are both expressions, they also can show up syntactically in places that for loop statements cannot, such as in lambda and object literals.

Because of all this, list comprehensions and map calls are worth knowing and using for simple sorts of iterations, especially if your application’s speed is an important consideration. For example, it’s hard to argue with the ease and power of code like either of the following:

[line.rstrip() for line in open('myfile')]
map((lambda line: line.rstrip()), open('myfile'))

To achieve the same memory economy and execution time division as map, though, list comprehensions must be coded as generator expressions—which is why we’ve run through this review in the first place, and one of the major topics this chapter turns to next.

State suspension

Unlike normal functions that return a value and exit, generator functions automatically suspend and resume their execution and state around the point of value generation. Because of that, they are often a useful alternative to both computing an entire series of values up front and manually saving and restoring state in classes. The state that generator functions retain when they are suspended includes both their code location and their entire local scope. Hence, their local variables retain information between results, and make it available when the functions are resumed.

The chief code difference between generator and normal functions is that a generator yields a value, rather than returning one—the yield statement suspends the function and sends a value back to the caller, but retains enough state to enable the function to resume from where it left off. When resumed, the function continues execution immediately after the last yield run. From the function’s perspective, this allows its code to produce a series of values over time, rather than computing them all at once and sending them back in something like a list.

Iteration protocol integration

To truly understand generator functions, you need to know that they are closely bound up with the notion of the iteration protocol in Python. As we’ve seen, iterator objects define a __next__ method, which either returns the next item in the iteration, or raises the StopIteration exception to end the iteration. An iterable object’s iterator is fetched initially with the __iter__ method, though this step is a no-op for objects that are their own iterator.

Python for loops, and all other iteration tools, use this iteration protocol to step through a sequence or value generator, if the protocol is supported (if not, iteration falls back on repeatedly indexing sequences instead). Any object that supports this interface works in all iteration tools, and the iter and next built-ins simplify manual iteration by calling the corresponding double-underscore method (or its internal equivalent).

To support this protocol, functions containing a yield statement are compiled specially as generators—they are not normal functions, but rather are built to return an object with the expected iteration-protocol methods. When later called, such functions return an iterable object that supports the iteration protocol with an automatically created method named __next__ to start or resume execution.

Besides yield, generator functions may also use a return statement that, along with falling off the end of the def block, terminates the generation of values by automatically raising a StopIteration exception. A generator’s return can also give an object that becomes the value attribute of the StopIteration exception, but it’s ignored by iteration tools and uncommon. From the caller’s perspective, the generator’s __next__ method simply starts or resumes the function and runs until either the next yield result is returned, or a StopIteration is raised.

The net effect is that generator functions, coded as def statements containing yield statements, are automatically made to support the iteration methods protocol and thus may be used in any iteration tool to produce results over time and on demand. The presence of a yield in a def suffices to make all this happen, and none of this applies to lambda because it doesn’t allow statements like yield (which is yet another reason to prefer def).

Generator functions in action

As usual, this is probably simpler in code than narrative. The following defines a generator function that can be used to produce the squares of a series of numbers—over time:

>>> def gensquares(n):
        for i in range(n):
            yield i ** 2        # Resume here later

This function yields a value, and so returns to its caller, each time through the loop. When it is resumed, its prior state is restored, including the last values of its variables i and n, and control picks up again immediately after the yield statement. For example, when it’s used in the body of a for loop, the first iteration starts the function and gets its first result; thereafter, control returns to the function after its yield statement each time through the loop:

>>> for i in gensquares(5):     # Resume the function
        print(i, end=' ')       # Print last yielded value

0 1 4 9 16

To end the generation of values, functions either use a return statement or simply allow control to fall off the end of the function body. As noted, return can give a value attached to the exit exception, but usually does not.

To most people, this process seems a bit implicit (if not enchanted) on first encounter. It’s actually quite tangible, though. If you really want to see what is going on inside the for, call the generator function directly:

>>> x = gensquares(3)           # Generator of 0..2 squares
>>> x
<generator object gensquares at 0x10dc6d700>

You get back a generator object that supports the iteration protocol—the generator function was compiled to return this automatically. The returned generator object in turn has a __next__ method that starts the function or resumes it from where it last yielded a value, and raises a StopIteration exception when the end of the series of values is reached and the function returns. As noted, next(X) here calls X.__next__() for convenience:

>>> next(x)                     # Run to first yield
0
>>> next(x)                     # Resume after yield, run to next yield
1
>>> next(x)                     # Resume after yield, run to next yield
4
>>> next(x)                     # Resume after yield, exception on return
StopIteration

Per Chapter 14, for loops and other iteration tools work with generators in the same way—by calling the __next__ method repeatedly, until the exit exception is caught. For generators, the result produces yielded values over time.

Notice that the top-level iter call of the iteration protocol isn’t required here because generators are their own iterator, supporting just one active iteration scan. To put that another way, generators return themselves for iter and support next directly. This also holds true in the generator expressions you’ll meet later in this chapter:

>>> y = gensquares(5)           # Returns a generator which is its own iterator
>>> iter(y) is y                # iter() is not required: a no-op here
True
>>> next(y)                     # Can run next() immediately
0

Why generator functions?

Given the simple example we’re using to illustrate fundamentals, you might be wondering just why you’d ever care to code a generator at all. In code so far, for instance, we could simply build the list of result values all at once:

>>> def buildsquares(n):
        res = []
        for i in range(n): res.append(i ** 2)
        return res

>>> for x in buildsquares(5): print(x, end=' ')

0 1 4 9 16

For that matter, we could use any of the for loop, map, or list comprehension techniques we’ve already mastered:

>>> for x in [n ** 2 for n in range(5)]:
        print(x, end=' ')

0 1 4 9 16

>>> for x in map((lambda n: n ** 2), range(5)):
        print(x, end=' ')

0 1 4 9 16

However, generators can be better in terms of both memory use and performance in larger programs. They allow functions to avoid doing all the work up front, which is especially useful when the result lists are large or when it takes a lot of computation to produce each value. Generators distribute the time required to produce the series of values among loop iterations, and can even produce a series of values that has no end at all (an “infinite” result).

Moreover, for more advanced uses, generators can provide a simpler alternative to manually saving the state between iterations in class objects—with generators, variables accessible in the function’s scopes are saved and restored automatically. You’ll be able to judge this contrast after we discuss class-based iterables in more detail in Part VI.

Generator functions are also more broadly focused than implied so far. They can operate on and return any type of object, and as iterables may appear in any of Chapter 14’s iteration tools, including tuple calls, enumerations, and dictionary comprehensions:

>>> def ups(line):
        for sub in line.split(','):          # Substring generator
            yield sub.upper()

>>> tuple(ups('aaa,bbb,ccc'))                # All iteration contexts
('AAA', 'BBB', 'CCC')

>>> {i: s for (i, s) in enumerate(ups('aaa,bbb,ccc'))}
{0: 'AAA', 1: 'BBB', 2: 'CCC'}

In a moment you’ll observe the same assets for generator expressions—a tool that trades function flexibility for comprehension conciseness. Later in this chapter you’ll also see that generators can sometimes enable otherwise impractical tasks, by producing components of result sets that would be far too large to create all at once. First, though, let’s explore some advanced generator function features.

Extended generator function protocol: send versus next

Somewhere along generators’ evolutionary path (in Python 2.5), a send method was added to the generator function protocol. The send method advances to the next item in the series of results, just like __next__, but also provides a way for the caller to communicate with the generator, and hence to affect its operation.

Technically, yield is an expression form that returns the item passed to send, not a statement. It can be coded either way (and usually is a statement), but when an expression must be enclosed in parentheses unless it’s the only item on the right side of an assignment statement. For example, X = yield Y is OK, as is X = (yield Y) + Z.

When this extra protocol is used, values are sent into a generator G by calling G.send(value). The generator’s code is then resumed, and the yield expression inside the generator returns the value passed to send by the caller. If the regular G.__next__() method (or its next(G) equivalent) is called to advance, the yield simply returns None. For example:

>>> def gen():
       for i in range(10):
           X = yield i
           print('=>', X)

>>> G = gen()
>>> next(G)              # Must call next() first, to start generator
0
>>> G.send(77)           # Advance, and send value to yield expression
=> 77
1
>>> G.send(88)
=> 88
2
>>> next(G)              # next() and X.__next__() send None
=> None
3

The send method can be used, for example, to code a generator that its caller can terminate by sending a termination code, or redirect by passing a new position in data being processed inside the generator.

In addition, generators also support a throw(type) method to raise an exception inside the generator at the latest yield, and a close method that raises a special GeneratorExit exception inside the generator to terminate the iteration entirely. Together with send, these are advanced features added to make generators more like a tool called coroutines, a role eventually subsumed in part by the upcoming async. Hence, we won’t delve further here; see Python’s standard manuals for more information, and watch for more on exceptions in Part VII.

Note, though, that while Python provides a next(X) convenience built-in that calls the X.__next__() method of an object, other generator methods, like send, must be called as methods of generator objects directly (e.g., G.send(X)). This makes sense if you realize that these extra methods are implemented on built-in generator objects only, whereas the __next__ method applies to all iterable objects—both built-in types and user-defined classes.

The yield from extension

Even later, Python 3.3 introduced extended syntax for the yield statement with a from generator clause, that allows generators to delegate to nested generators (known as subgenerators). In simple cases, it’s the equivalent to a yielding for loop. As a demo, the list call in the following forces a generator to produce values from two sources:

>>> def both(N):
        for i in range(N): yield i
        for i in map(lambda x: x ** 2, range(N)): yield i

>>> list(both(5))
[0, 1, 2, 3, 4, 0, 1, 4, 9, 16]

The yield from syntax makes this more concise and explicit, and supports all the usual generator usage contexts. In the following, both is called a delegating generator, and the range and map built-ins serve as subgenerators:

>>> def both(N):
        yield from range(N)
        yield from map(lambda x: x ** 2, range(N))

>>> list(both(5))
[0, 1, 2, 3, 4, 0, 1, 4, 9, 16]

>>> ' : '.join(str(i) for i in both(5))
'0 : 1 : 2 : 3 : 4 : 0 : 1 : 4 : 9 : 16'

The yield from also supports arbitrary chains of generators. In the following, for example, two generators yield from other, the last of which ultimately uses yield to send results up though the chain:

>>> def c(n):
        yield n * 8                   # The bottom of the "chain"
        yield n ** 2

>>> def b(n):
        yield from c(n)               # Delegate to subgenerator
 
>>> def a(n):
        yield from b(n)               # Delegate to subgenerator

>>> g = a(4)                          # Make top generator, fetch results
>>> g
<generator object a at 0x10bd17940>
>>> next(g)
32

>>> for i in a(4): print(i)           # Force results production
32
16

In more advanced roles, though, this extension allows subgenerators to receive sent and thrown values directly from the delegating generator’s caller, and return a final value to the delegating generator as the result of the yield from expression. The net effect allows generators to be split into multiple subgenerators much as a single function can be split into multiple subfunctions, but still operate as though their code appeared inline where the yield from appears.

Since this is both uncommon and beyond this chapter’s scope, we’ll again defer to Python’s standard manuals for more details. For an additional yield from example, see the solution to this part’s Exercise 11 described at the end of Chapter 21, and stay tuned for a glimpse of async def coroutines ahead—whose await is something like a generator yield from, with a delegation to an event loop to allow other tasks to be run during pauses. Here, let’s move on to a tool close enough to yield to be called a fraternal twin.

Why generator expressions?

Just like generator functions, generator expressions are a memory-space optimization—they do not require the entire result list to be constructed all at once, as the square-bracketed list comprehension does. Also like generator functions, they divide the work of results production into smaller time slices—they yield results in piecemeal fashion, instead of making the caller wait for the full set to be created in a single call.

On the other hand, because generator expressions often run slower today than list comprehensions, they may be best reserved for result sets that are very large, or programs that cannot wait for full results generation. A more authoritative statement about performance, though, will have to await the timing scripts we’ll code in the next chapter.

On the subjective upside, generator expressions also offer significant coding advantages—as the next sections show.

Generator expressions versus map

One way to see the coding benefits of generator expressions is to compare them to other functional tools, as we did for list comprehensions. For example, generator expressions often are equivalent to map calls, because both generate result items on request. Like list comprehensions, though, generator expressions may be simpler to code when the operation applied is not a function call:

>>> list(map(abs, (-1, -2, 3, 4)))                        # Map function on tuple
[1, 2, 3, 4]
>>> list(abs(x) for x in (-1, -2, 3, 4))                  # Generator expression
[1, 2, 3, 4]

>>> list(map(lambda x: x * 2, (1, 2, 3, 4)))              # Nonfunction case
[2, 4, 6, 8]
>>> list(x * 2 for x in (1, 2, 3, 4))                     # Simpler as generator?
[2, 4, 6, 8]

The same holds true for text-processing use cases like the join call we saw earlier—a list comprehension makes an extra temporary list of results, which is completely pointless in this context because the list is not retained, and map loses simplicity points compared to generator expression syntax when the operation being applied is not a call:

>>> line = 'aaa,bbb,ccc'
>>> ''.join([x.upper() for x in line.split(',')])         # Makes temporary list
'AAABBBCCC'

>>> ''.join(x.upper() for x in line.split(','))           # Generates results
'AAABBBCCC'
>>> ''.join(map(str.upper, line.split(',')))              # Generates results
'AAABBBCCC'

>>> ''.join(x * 2 for x in line.split(','))               # Simpler as generator?
'aaaaaabbbbbbcccccc'
>>> ''.join(map(lambda x: x * 2, line.split(',')))
'aaaaaabbbbbbcccccc'

Both map and generator expressions can also be arbitrarily nested; when coded this way, a list call or other iteration tool starts the entire process of producing results. For example, the list comprehension in the following produces the same result as the map and generator equivalents that follow it, but makes two physical lists. The others generate just one integer at a time with nested generators, and the generator expression form may more clearly reflect its intent:

>>> [x * 2 for x in [abs(x) for x in (-1, -2, 3, 4)]]        # Nested list comps
[2, 4, 6, 8]

>>> list(map(lambda x: x * 2, map(abs, (-1, -2, 3, 4))))     # Nested maps
[2, 4, 6, 8]

>>> list(x * 2 for x in (abs(x) for x in (-1, -2, 3, 4)))    # Nested generators
[2, 4, 6, 8]

Although the effect of all three of these is to combine operations, the generators do so without making multiple temporary lists. In fact, the next example both nests and combines generators—the nested generator expression is activated by map, which in turn is only activated by list:

>>> import math
>>> list(map(math.sqrt, (x ** 2 for x in range(4))))         # Nested combos
[0.0, 1.0, 2.0, 3.0]

Technically speaking, the range on the right in the preceding is a value generator too, activated by the generator expression itself—forming three levels of value generation, which produce individual values from inner to outer only on request, and which “just works” because of Python’s iteration tools and protocol. In fact, generator nestings can be arbitrarily mixed and deep, though some may be more valid than others:

>>> list(map(abs, map(abs, map(abs, (-1, 0, 1)))))           # Nesting gone bad?
[1, 0, 1]
>>> list(abs(x) for x in (abs(x) for x in (abs(x) for x in (-1, 0, 1))))
[1, 0, 1]

These last examples illustrate how general generators can be, but are also coded in an intentionally complex form to underscore that generator expressions have the same potential for abuse as the list comprehensions discussed earlier—as usual, you should keep them simple unless they must be complex.

When used well, though, generator expressions combine the expressiveness of list comprehensions with the space and time benefits of other iterables. The following nonnested alternatives, for example, provide simpler solutions to the preceding three listings’ nested codings, but still leverage generators’ strengths:

>>> list(abs(x) * 2 for x in (-1, -2, 3, 4))                 # Same, unnested
[2, 4, 6, 8]
>>> list(math.sqrt(x ** 2) for x in range(4))                # Flat may be better
[0.0, 1.0, 2.0, 3.0]
>>> list(abs(x) for x in (-1, 0, 1))
[1, 0, 1]

Generator expressions versus filter

Generator expressions also support all the usual list comprehension syntax—including if clauses, which work like the filter call we met earlier. Because filter is an iterable that generates its results on request, a generator expression with an if clause is operationally equivalent. Again, the join in the following is an iteration tool that suffices to force all forms to produce their results:

>>> line = 'aa bbb c'
>>> ''.join(x for x in line.split() if len(x) > 1)           # Generator with if
'aabbb'
>>> ''.join(filter(lambda x: len(x) > 1, line.split()))      # Equivalent filter
'aabbb'

The generator seems just marginally simpler than the filter here. As for list comprehensions, though, adding processing steps to filter results requires a map too, which makes filter noticeably more complex:

>>> ''.join(x.upper() for x in line.split() if len(x) > 1)
'AABBB'
>>> ''.join(map(str.upper, filter(lambda x: len(x) > 1, line.split())))
'AABBB'

In effect, generator expressions provide more general coding structures that do not rely on functions, but still delay results production. Also like list comprehensions, there is always a statement-based equivalent to a generator expression, though it sometimes renders substantially more code:

>>> res = ''
>>> for x in line.split():                              # Statement equivalent?
        if len(x) > 1:                                  # This is also a join
            res += x.upper()

>>> res
'AABBB'

In this case, though, the statement form isn’t quite the same—it cannot produce items one at a time, and it’s also emulating the effect of the join that forces results to be produced all at once. The true equivalent to a generator expression would be a generator function with a yield, as the next section will show.

Generators are single-pass iterables

First up, a subtle but important point: both generator functions and generator expressions are their own iterators and thus support just one active iteration—unlike some built-in types, you can’t have multiple iterators of either generator positioned at different locations in the stream of results. Because of this, a generator’s iterator is the generator itself; in fact, as suggested earlier, calling iter on a generator expression or function is an optional no-op:

>>> G = (c * 4 for c in 'hack')
>>> iter(G) is G                          # My iterator is myself: G has __next__
True

If you do iterate over the results stream manually with multiple iterators, they will all point to the same position:

>>> G = (c * 4 for c in 'hack')           # Make a new generator
>>> I1 = iter(G)                          # Iterate manually
>>> next(I1)
'hhhh'
>>> next(I1)
'aaaa'
>>> I2 = iter(G)                          # Second iterator at same position!
>>> next(I2)
'cccc'

Moreover, once any iteration runs to completion, all are exhausted—we have to make a new generator to start again:

>>> list(I1)                              # Collect the rest of I1's items
['kkkk'] 
>>> next(I2)                              # Other iterators exhausted too
StopIteration

>>> I3 = iter(G)                          # Ditto for new iterators
>>> next(I3)
StopIteration

>>> I3 = iter(c * 4 for c in 'hack')      # New generator to start over
>>> next(I3)
'hhhh'

The same holds true for generator functions—the following def statement-based equivalent supports just one active iterator and is exhausted after one pass:

>>> def timesfour(S):
        for c in S:
            yield c * 4

>>> G = timesfour('hack')                 # Generator functions work the same way
>>> iter(G) is G                          # One scan per generator (call)
True
>>> I1, I2 = iter(G), iter(G)
>>> next(I1)
'hhhh'
>>> next(I1)
'aaaa'
>>> next(I2)                              # I2 at same position as I1
'cccc'

This is different from the behavior of some built-in objects like lists, which support multiple iterators and passes and even reflect their in-place changes in active iterators:

>>> L = [1, 2, 3, 4]
>>> I1, I2 = iter(L), iter(L)
>>> next(I1)
1
>>> next(I1)
2
>>> next(I2)                              # Lists support multiple iterators
1
>>> del L[2:]                             # Changes reflected in iterators
>>> next(I1)
StopIteration

Though not readily apparent in these simple examples, this can matter in your code: if you wish to scan a generator’s values multiple times, you must either create a new generator for each scan or build a rescannable list out of its values—a single generator’s values will be consumed and exhausted after a single pass. See this chapter’s sidebar “Why You Will Care: Iteration Versus Python Morph” for a prime example of the sort of code impacted by this.

When we begin coding class-based iterables in Part VI, you’ll also see that it’s up to us to decide how many iterations we wish to support for our objects, if any. In general, objects that wish to support multiple scans will return supplemental class objects instead of themselves for iter. The next section previews more of this model.

Generation in built-ins and classes

Although we’ve focused on coding value generators ourselves in this section, don’t forget that many built-in types behave the same way. As we saw in Chapter 14, for example, dictionaries and files generate results too:

for key in dictionary: …                          # See Chapter 14 et al.
for line in file: …

Though beyond this book’s scope, many Python standard library tools generate values too, including its directory walker—which at each level of a folder tree yields a tuple of the current directory, its subdirectories, and its files:

>>> import os
>>> for (root, subs, files) in os.walk('..'):     # Directory-walk generator
        for name in files:                        # A Python 'find' operation
            if name.endswith('.py'):              # Also os.path.join(root, name)
                print(root, name)

../Chapter02 script0.py
../Chapter03 myfile.py
../Chapter03 script1.py
…etc…

Because the current Python-coded implementation of os.walk uses yield to return results, it’s a normal generator function, and hence iterable object, that generates a three-item tuple on each iteration:

>>> G = os.walk('..')       # A single-scan iterator: iter(G) optional
>>> next(G)
('..', ['Chapter02', 'Chapter03', 'Chapter04', …etc… 'Chapter37'], [])
>>> next(G)
('../Chapter02', ['__pycache__'], ['script0.py'])

By yielding results as it goes, the walker does not require its clients to wait for an entire tree to be scanned. See Python’s manuals for more on this tool. For system-tools fans, also see the next chapter for an example that uses os.popen—a related iterable used to run a shell command and read its output. And for additional examples of built-in value generators and the iteration tools that use them, review Chapter 14.

Finally, although beyond the scope of this chapter, it is also possible to implement arbitrary user-defined generator objects with classes that conform to the iteration protocol. Such classes define __iter__ and __next__ methods explicitly to support the protocol:

class SomeIterable:
    def __iter__(...): ...     # On iter(): return self or supplemental object
    def __next__(...): ...     # On next(): coded here, or in another class

As the prior section suggested, these classes usually return their objects directly for single-iteration behavior, or a supplemental object with scan-specific state for multiple-scan support.

Alternatively, a user-defined iterable class’s __iter__ may use yield to transform itself into a generator, and a __getitem__ indexing method is also available as a fallback option for iteration with trade-offs we’ll explore later. However this is coded, the iterator and generator story won’t really be complete until we’ve seen how it also maps to classes. For now, we’ll have to settle for postponing its conclusion—and its final sequel—until Chapter 30.

Comprehensions versus type calls and generators

We’ve also been focusing on list comprehensions and generators in this chapter, but keep in mind that there are two other comprehension expression forms: the set and dictionary comprehensions we met earlier in Chapter 14 and Part II. For reference and closure, here’s a summary of all the comprehension forms in Python:

>>> [x * x for x in range(10)]               # List comprehension: builds list
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]         # Fixed order, made all at once

>>> (x * x for x in range(10))               # Generator expression: yields items
<generator object <genexpr> at 0x100a1f920>

>>> {x * x for x in range(10)}               # Set comprehension: builds set
{0, 1, 64, 4, 36, 9, 16, 49, 81, 25}         # random order, no duplicates

>>> {x: x * x for x in range(10)}            # Dict comprehension: insert order
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

All of these forms use the same formal comprehension syntax we met earlier, but their enclosing delimiters (and key, for dictionaries) denotes their differing roles. In a sense, list, set, and dictionary comprehensions are syntactic sugar for passing generator expressions to type names. Because all accept any iterable, a generator works here too:

>>> list(x ** 2 for x in range(10))          # Generator plus type name
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> set(x * x for x in range(10))            # Similar to a set comprehension
{0, 1, 64, 4, 36, 9, 16, 49, 81, 25}

>>> dict((x, x * x) for x in range(10))
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

But don’t read too much into such equivalences. Because implementations may vary arbitrarily, you should generally collect performance data before adopting an alternative. In this case, generators are actually slower than the equivalent list comprehension today, as we’ll prove in the next chapter. Programs that run lots of them may need to care.

Scopes and comprehension variables

Now that we’ve seen all comprehension forms, Chapter 17’s overview of the localization of loop variables in these expressions may make more sense. In all forms, the loop variable (or variables) after the for is local the expression—it won’t clash with names outside, but is also not available there, and this differs from for statements:

>>> X = 99
>>> [X for X in range(5)]         # All comprehensions localize loop variables
[0, 1, 2, 3, 4]
>>> X                             # Enclosing-scope X was not changed
99
>>> for X in range(5): pass       # But loop statements do not localize!
>>> X
4

As noted in Chapter 17, loop variables assigned in a comprehension really live in a further nested special-case scope, but other names referenced within these expressions follow the usual LEGB rules. In the following generator, for example, Z is localized in the comprehension, but Y and X are found in the enclosing local and global scopes as usual:

>>> X = 'aaa'
>>> def func():
        Y = 'bbb'
        print('-'.join(Z for Z in X + Y))    # Z=comprehension, Y=local, X=global

>>> func()
a-a-a-b-b-b

One exception here: names assigned by the := expression inside a comprehension do leak out of comprehension, and generally behave as though they were assigned in the scope containing the comprehension itself:

>>> S = 'hack'
>>> [(temp := S[i]) + temp.upper() for i in range(len(S))]
['hH', 'aA', 'cC', 'kK']
>>> temp
'k'
>>> i
NameError: name 'i' is not defined. Did you mean: 'id'?

While this allows comprehensions to both reuse and export values, keep in mind that a lambda container’s local scope effectively plugs the leak—temp in the following, for example, lives only in the lambda’s scope:

>>> del temp
>>> f = lambda: [(temp := S[i]) + temp.upper() for i in range(len(S))]
>>> f()
['hH', 'aA', 'cC', 'kK']
>>> temp
NameError: name 'temp' is not defined

We’ll code such a := nesting in a sequence-scrambler example ahead, though some observers may deem it much less transparent than simple assignment statements, and other real-world roles must await your discovery. See Chapter 11 for more background on := named assignment.

Generating “infinite” (well, indefinite) results

Finally, it was noted earlier in passing that generators can even produce “infinite” results that tools like list comprehensions cannot. This may sound more impressive that it is; really, this just means that a generator can keep yielding results from its retained state’s local variables indefinitely—until its client grows tired of them:

>>> def squares(n):
        while True:                      # Generate results "forever"
            yield n ** 2                 # Or until no more next calls...
            n += 1

>>> G = squares(2)
>>> next(G)
4
>>> next(G)
9
>>> for i in range(10): print(next(G), end=' ')

16 25 36 49 64 81 100 121 144 169

This pattern may be useful in limited roles like test-parameter generation, and other tools can’t compete in this event; list comprehensions, for example, must collected all results at once. In the end, though, this is a narrow role, generators are mortal, and so are we—so let’s move to some code that’s a bit more tangible in the next section.

Simple functions

As is, this code works on specific named variables only, and simply prints its result. As we’ve seen, it’s easy to generalize this by turning it into a normal function that can both work on any object passed to its argument, and return its result for arbitrary use. The following does so for the second coding alternative; since its first cut exhibits the classic list comprehension pattern, we can also save some work by coding it as such in the second:

>>> def scramble(seq):
        res = []
        for i in range(len(seq)):
            res.append(seq[i:] + seq[:i])
        return res

>>> scramble('code')
['code', 'odec', 'deco', 'ecod']

>>> def scramble(seq):
        return [seq[i:] + seq[:i] for i in range(len(seq))]

>>> scramble('code')
['code', 'odec', 'deco', 'ecod']

>>> for x in scramble((1, 2, 3)):
        print(x, end=' ')

(1, 2, 3) (2, 3, 1) (3, 1, 2)

We could use recursion here as well, but it’s probably overkill in this fixed and linear context.

Generator functions

The preceding section’s simple approach works, but must build an entire result list in memory all at once (not great on memory usage if it’s massive), and requires the caller to wait until the entire list is complete (less than ideal if this takes a substantial amount of time). You should know by now that we can do better on both fronts by translating this to a generator function that yields one result at a time, using either coding scheme:

>>> def scramble(seq):
        for i in range(len(seq)):
            seq = seq[1:] + seq[:1]               # Generator function, coding 1
            yield seq                             # Assignments work here

>>> def scramble(seq):
        for i in range(len(seq)):                 # Generator function, coding 2
            yield seq[i:] + seq[:i]               # Yield one item per iteration

Both of these alternatives produce values when used by an iteration tools like list or for, though the second version’s results order (shown here) again varies slightly in ways that probably won’t matter to future clients:

>>> list(scramble('code'))                        # list() generates all results
['code', 'odec', 'deco', 'ecod'] 

>>> list(scramble((1, 2, 3)))                     # Any sequence type works
[(1, 2, 3), (2, 3, 1), (3, 1, 2)]

>>> for x in scramble((1, 2, 3)):                 # for loops generate results
        print(x, end=' ')

(1, 2, 3) (2, 3, 1) (3, 1, 2)

Both generator functions retain their local scope state (seq and i) while active, minimize memory space requirements, and divide the work into shorter time slices. As full functions, they are also very general. Moreover, because iteration tools work the same whether stepping through a real list or a generator of values, the function can select between the two codings freely, and even change strategies in the future without impacting callers.

Generator expressions

As we’ve also seen, generator expressions—comprehensions in parentheses instead of square brackets—also generate values on request and retain their local state. As expressions, they’re not as flexible as full functions, but because they yield their values automatically, generator expressions can often be more concise in specific use cases like this:

>>> S = 'code'
>>> G = (S[i:] + S[:i] for i in range(len(S)))         # Generator expr, coding 2
>>> list(G)
['code', 'odec', 'deco', 'ecod']

A generator expression can’t use the seq assignment statement of the first generator-function coding, but it can achieve the same effect by nesting the := named-assignment expression covered in Chapter 11, because := both changes assigned names in the containing scope, and returns the assigned value so it appears in the results stream:

>>> S = 'code'
>>> G = (S:=(S[1:] + S[:1]) for i in range(len(S)))    # Generator expr, coding 1
>>> list(G)
['odec', 'deco', 'ecod', 'code']

Either way, we’re still operating on a specific variable here, S. To generalize a generator expression for an arbitrary subject, wrap it in a simple lambda function that takes an argument and returns a generator that uses it (subtle bit: note that seq in the second of these is not local to the generator, but is local to the lambda that encloses it):

>>> F = lambda seq: (seq[i:] + seq[:i] for i in range(len(seq)))
>>> list(F(S))
['code', 'odec', 'deco', 'ecod']

>>> F = lambda seq: (seq:=(seq[1:] + seq[:1]) for i in range(len(seq)))
>>> list(F(S))
['odec', 'deco', 'ecod', 'code'] 

>>> list(F([1, 2, 3]))
[[2, 3, 1], [3, 1, 2], [1, 2, 3]] 

>>> F(S)
<generator object <lambda>.<locals>.<genexpr> at 0x100b23300>

>>> for x in F((1, 2, 3)): print(x, end=' ')
(2, 3, 1) (3, 1, 2) (1, 2, 3)

Tester client

Finally, we can use either the generator function or its expression equivalent in Chapter 18’s tester to produce scrambled arguments. As packaged in the module of Example 20-1, the sequence scramblers become reusable tools.

"Generate shuffles of a sequence, by function or expression"

def scramble1(seq):
    for i in range(len(seq)):                
        yield seq[i:] + seq[:i]        # Yield one shuffle per iteration

scramble2 = lambda seq: (seq[i:] + seq[:i] for i in range(len(seq)))

Though it requires a bit of page flipping to see how, moving the values generation out to an external tool like this also makes the testing function noticeably simpler:

>>> from scramble import scramble1           # Choose your scrambler
>>> from inter2 import intersect, union      # Functions to be tested

>>> def tester(func, items, trace=True):
        for args in scramble1(items):        # Use either generator
            if trace: print(args)
            print(sorted(func(*args)))       # Test for this scramble

>>> tester(intersect, ('aab', 'abcde', 'ababab'), False)
['a', 'b']
['a', 'b']
['a', 'b']

To make this work for yourself, make sure all imported files are in the current directory: either copy inter2.py from Chapter 18’s folder to Chapter 20’s, or copy Example 20-1’s scramble.py to Chapter 18’s folder and work there (the examples package has already done the former). Alternatively, you can modify the module import search path’s PYTHONPATH setting to include any folder—as you’ll learn when we cover modules in full after the next chapter.

Why generators here: Space, time, and more

Generators are a somewhat advanced tool, and might be better treated as an optional topic, but for the fact that they permeate the Python language today. As we’ve seen, fundamental built-in tools such as range, map, dictionary keys, and even files are now generators, so you must be familiar with the concept even if you don’t write new generators of your own. Moreover, user-defined generators are increasingly common in Python code that you might come across today—in the Python standard library, for instance.

Though your code is yours to code, the same guidelines given for list comprehensions apply here as well: don’t complicate your code with user-defined generators if they are not warranted. Especially for smaller programs and data sets, such tools may not make sense. Simple lists of results may suffice, may be easier to understand, will be garbage-collected automatically, and might be produced quicker (and are today: see the next chapter). Advanced tools like generators that rely on implicit “magic” can be fun to experiment with, but may be subpar when optional.

That being said, there are specific use cases that generators can address well. They can reduce memory footprint in some programs, reduce delays in others, and can occasionally make the impossible possible. Consider, for example, a program that must produce all possible permutations of a nontrivial sequence. Since the number of combinations is a factorial that explodes exponentially, the preceding permute1 recursive list-builder function will either introduce a noticeable and perhaps interminable pause, or fail completely due to memory requirements.

By contrast, the permute2 recursive generator returns each individual result quickly, and can handle very large result sets. Even at just 10 items, for instance, the difference starts becoming stark (per Python’s math module):

>>> import math
>>> f'{math.factorial(10):,}'        # 10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1
'3,628,800'

>>> from permute import permute1, permute2
>>> seq = list(range(10))

>>> p1 = permute1(seq)               # ~17 seconds on a 2.3GHz i9 macOS host
>>> len(p1)                          # Creates a list of 3.6M numbers
3628800
>>> p1[0], p1[-1]
([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [9, 8, 7, 6, 5, 4, 3, 2, 1, 0])

In this case, the permute1 list builder pauses for 17 seconds to build a 3.6-million-item list, but the generator can begin returning individual results immediately—neither permute2 nor any next pause in the following:

>>> p2 = permute2(seq)               # Returns generator immediately
>>> next(p2)                         # Returns each result quickly on demand
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> next(p2)
[0, 1, 2, 3, 4, 5, 6, 7, 9, 8]

While collecting all results from the generator is still slow, it’s faster than the list builder, and not intended usage:

>>> p2 = list(permute2(seq))         # About 10 seconds, though still impractical
>>> p1 == p2                         # Same set of results generated
True

Naturally, we might be able to optimize the list builder’s code to run quicker (e.g., an explicit stack instead of recursion might change its performance), but for larger sequences, it’s not an option at all—at just 50 items, the number of permutations wholly precludes building a results list, and would take far too long for mere mortals like us in any event (and larger values will overflow the preset recursion stack depth limit: see the preceding chapter). The generator, however, is still viable—despite the factorial size of the problem, it is able to produce individual results immediately:

>>> math.factorial(50)
30414093201713378043612608166064768844377641568960512000000000000

>>> p3 = permute2(list(range(50)))       # permute1 is not an option here!
>>> next(p3)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49]

For more fun—and to yield results that are more variable and less obviously deterministic—we could use Python’s random module of Chapter 5 to randomly shuffle the sequence to be permuted before the permuter begins its work. In the following, for example, each permute2 and next call returns immediately as before and permutes a random sequence, but permute1 hangs (and presumably perishes from memory starvation if allowed to run long enough):

>>> seq = list(range(20))
>>> print(f'{math.factorial(20):,}')     # permute1 is not an option here
2,432,902,008,176,640,000

>>> import random
>>> random.shuffle(seq)                  # Shuffle sequence randomly first
>>> p = permute2(seq)
>>> next(p)
[10, 19, 5, 6, 2, 13, 1, 8, 11, 7, 14, 16, 4, 3, 0, 18, 9, 12, 17, 15]
>>> next(p)
[10, 19, 5, 6, 2, 13, 1, 8, 11, 7, 14, 16, 4, 3, 0, 18, 9, 12, 15, 17]

>>> random.shuffle(seq)
>>> p = permute2(seq)
>>> next(p)
[17, 15, 14, 7, 10, 8, 2, 6, 18, 19, 13, 4, 1, 12, 5, 0, 3, 9, 16, 11]
>>> next(p)
[17, 15, 14, 7, 10, 8, 2, 6, 18, 19, 13, 4, 1, 12, 5, 0, 3, 9, 11, 16]

In fact, we might be able to use the random shuffler instead of manual shuffles in some roles, as long as we either can assume that it won’t repeat shuffles during the time we consume them, or test its results against prior shuffles to avoid repeats—and hope that we do not live in the strange universe where a random sequence repeats the same result an infinite number of times! When full coverage is important, manual shuffles offer better control.

The main point here is that generators can sometimes produce results from large solution sets when list builders cannot. Then again, it’s not clear how common such use cases may be in the real world, and this doesn’t necessarily justify the implicit flavor of value generation that we get with generator functions and expressions. As you’ll see in Part VI, value generation can also be coded as iterable objects with classes. Class-based iterables can produce items on request too, and are more explicit than the magic objects and methods produced for generator functions and expressions.

Part of programming is finding a balance among trade-offs like these, and there are no absolute rules here. While the benefits of generators may sometimes justify their use, maintainability counts too. Like comprehensions, generators also offer an expressiveness and code economy that’s hard to resist if you understand how they work—but you’ll want to weigh this against the frustration of coworkers who might not.

Running serial tasks with normal blocking calls

As usual, this may be easier to grok in code than narrative, so let’s turn to some examples to demo the salient bits. All of this section’s example snippets live in file all_async_demos.py in the examples package, previewed in Example 20-7. To work along, ether cut and paste from this file or emedia, or run the examples file in its entirety or with parts stubbed out with triple quotes. All examples here assume that the preamble at the top of Example 20-7 has been run.

import time, asyncio
def now(): 
    return time.strftime('[%H:%M:%S]')      # Local time, as hour:minute:second

…all examples here…

We need a handful of standard library tools here: some in the asyncio standard library module are required, and we’ll display times with time.strftime and pause with sleep calls in both time and asyncio to simulate a long-running task. See Python’s library manual for these tools’ fine print that we’ll skip here for space and scope.

To get started, consider the following non-async code, in which a main function calls another, producer, that simply pauses for two seconds using the time.sleep call before returning a result. The sleep stands in for a real blocking operation—like an IO call, network transfer, or user interaction. As usual with normal functions, main simply waits for each producer call to run to completion and return a result, before it proceeds with a next step:

def producer(label):
    time.sleep(2)                              # Pause for two seconds: blocking
    return f'All done, {label}, {now()}'       # And return a result

def main():
    print('Start =>', now())
    print(producer(f'serial task 1'))          # Run three steps in sequence
    print(producer(f'serial task 2'))          # Waiting for each one to finish
    print(producer(f'serial task 3'))          # Before doing anything else
    print('Stop  =>', now())

main()

When run, this code produces the following mundane results. As you can see, each producer call runs a two-second sleep before another starts, so the total time required is six seconds—two for each call run in series by main:

Start => [19:00:28]
All done, serial task 1, [19:00:30]
All done, serial task 2, [19:00:32]
All done, serial task 3, [19:00:34]
Stop  => [19:00:34]

Running concurrent tasks with “await” and “async def”

Now, let’s do the same work with async functions. In the next listing, the producer is coded as an async def function, which makes Python compile it specially: it’s now a coroutine—an object which, when called, doesn’t produce its result, but instead returns an automatically created object that supports the coroutine method protocol. That protocol generally includes a method named __await__, which returns an iterator that produces coroutine results on __next__, but may instead use __anext__ for generators. We can safely ignore all of these here.

The async def syntax is always required for functions that use await, as well as aync for and async with demoed ahead. Because async def is required to both use and define awaitable coroutines, it tends to spread viruslike throughout code (in fact, some tools now come with libraries in two separate forms, asynchronous and not).

Also in the following, producer uses await to suspend itself until the sleep expires. await is similar to a yield from, but the event loop is free to pause the coroutine running the await and resume another if the awaited result is not ready. Notice that await is used as a statement in producer; it’s also an expression that returns the awaited result, as in main, where it returns the result of the awaited objects:

async def producer(label):                     # await requires async
    await asyncio.sleep(2)                     # Call nonblocking/awaitable sleep
    return f'All done, {label}, {now()}'       # Result of await expression

async def main():
    print('Start =>', now())
    task1 = asyncio.create_task(producer(f'async task 1'))
    task2 = asyncio.create_task(producer(f'async task 2'))
    task3 = asyncio.create_task(producer(f'async task 3'))
    print(await task1) 
    print(await task2) 
    print(await task3)                         # Wait for tasks to finish
    print('Stop  =>', now())

asyncio.run(main())                            # Start event-loop schedule

Crucially, the awaited asyncio.sleep call is like time.sleep, but is both nonblocking—it uses the event-loop’s API to ensure that the caller doesn’t pause for its completion, and awaitable—it uses async def to allow itself to be paused until a timeout elapses. Operations at the end of await chains must generally follow this model.

Also crucially, main uses asyncio.create_task to wrap each producer object in a task—an object run by the event loop that suspends execution of the coroutine it wraps while the coroutine waits for completion of a future, and returns the result of its wrapped coroutine when awaited itself. Future objects signal progress in ways that are too low level for this overview; here, it’s enough to know that each task suspends itself while its coroutine is sleeping.

As a result, tasks can be overlapped in time, to run concurrently. In our code, all three tasks—and the producer coroutines they manage—finish at the same time, and the total program takes just two seconds instead of six:

Start => [19:00:34]
All done, async task 1, [19:00:36]
All done, async task 2, [19:00:36]
All done, async task 3, [19:00:36]
Stop  => [19:00:36]

Naturally, we can use loops here to make the steps more automated. The following runs the same as the prior version—taking 2 total seconds, and two for each overlapped task:

async def producer(label):
    await asyncio.sleep(2)
    return f'All done, {label}, {now()}'

async def main():
    print('Start =>', now())
    tasks = []
    for i in range(3):
        tasks.append(asyncio.create_task(producer(f'async task {i+1}')))
    for task in tasks:
        print(await task)
    print('Stop  at', now())

asyncio.run(main())

In both producer and main, an await, much like a yield from, suspends execution of the coroutine that runs it until the awaited object returns its result. Together with the event loop and tasks, the net effect interleaves code.

How not to use async functions

To understand the preceding example, it may help to see other codings that do not work. In this example, the awaits, tasks, and event-loop are all essential: the code won’t work without them. For one thing, calling main directly without routing it to the event loop fails right out of the gate with a warning. The call to main simply creates an awaitable object but does noting with it (for brevity, snippets in this section show just differing code, followed by its results):

async def main():
    print(producer('xxx'))

main()

# Result…
…/all_async_blunders.py:12: RuntimeWarning: coroutine 'main' was never awaited

For another, calling producer directly instead of awaiting it fails too, and for the same reason. Much like generators, calling a coroutine returns the awaitable object, but doesn’t run it; we must await the awaitable to make it go, just like we have to run next to get results from a generator:

async def main():
    print(producer('xxx'))

asyncio.run(main())

# Result…
<coroutine object producer at 0x10142e740>
…/all_async_blunders.py:22: RuntimeWarning: coroutine 'producer' was never awaited

More vitally, tasks are key to the concurrency here. If the example’s main simply runs an await for a producer directly, it will pause for the sleep too. In the following miscoding, both coroutines await the sleep’s end, and the two producers run serially—yielding a total of four seconds for the program at large instead of two:

async def main():
    print('Start =>', now())
    print(await producer('xxx'))
    print(await producer('yyy'))
    print('Stop  =>', now())

asyncio.run(main())

# Result…
Start => [18:41:56]
All done, xxx, [18:41:58]
All done, yyy, [18:42:00]
Stop  => [18:42:00]

This is also true if we defer the awaits by assignments—main continues to run after creating the producer coroutine awaitables, but still hangs for each later await that runs the coroutine’s code, with no concurrency to be found:

async def main():
    print('Start =>', now())
    p1 = producer('xxx')
    p2 = producer('yyy')
    print('Await =>', now())
    print(await p1)
    print(await p2)
    print('Stop  =>', now())

asyncio.run(main())                            

# Result…
Start => [18:42:00]
Await => [18:42:00]
All done, xxx, [18:42:02]
All done, yyy, [18:42:04]
Stop  => [18:42:04]

That’s why tasks are needed in the prior example: they enable coroutines to be paused and run by the event loop to overlap their execution. That is, they enable concurrency—along with alternatives like those of the next section.

Running concurrent tasks with “as_completed” and “gather”

But enough common mistakes: let’s get back to coding concurrent coroutines the right way. Besides creating tasks, we can use the gather method to wait for all of our coroutines automatically, or as_completed to wait for results as they finish. The latter can be used in a normal for to watch for completions:

async def producer(label):
    await asyncio.sleep(2)
    return f'All done, {label}, {now()}'

async def main():
    print('Start =>', now())
    coros = [producer(f'async task {i+1}') for i in range(3)]
    for nextdone in asyncio.as_completed(coros):
        print(await nextdone)
    print('Stop  at', now())

asyncio.run(main())

This also finishes in two total seconds, though coroutine results may filter in in any order (as_completed can also be used with async for of the next section in ways covered by Python’s manuals):

Start => [19:00:38]
All done, async task 2, [19:00:40]
All done, async task 3, [19:00:40]
All done, async task 1, [19:00:40]
Stop  at [19:00:40]

Alternatively, the gather call schedules tasks for coroutines, awaits all its arguments, and returns a list of task return values in the same order as its arguments when all have finished:

async def producer(label):
    await asyncio.sleep(2)
    return f'All done, {label}, {now()}'

async def main():
    print('Start =>', now())
    coro1 = producer(f'async task 1')
    coro2 = producer(f'async task 2')
    coro3 = producer(f'async task 3')
    results = await asyncio.gather(coro1, coro2, coro3)
    print(results) 
    print('Stop  at', now())

asyncio.run(main())

Result vary because they are in a list (formatted for display here), but it’s still just two seconds for the entire gig:

Start => [19:00:40]
['All done, async task 1, [19:00:42]', 
 'All done, async task 2, [19:00:42]', 
 'All done, async task 3, [19:00:42]']
Stop  at [19:00:42]

For any line counters in the audience, a starred list comprehension can be used to create awaitables for gather more succinctly, and produces the same two-second result as the preceding version:

async def producer(label):
    await asyncio.sleep(2)
    return f'All done, {label}, {now()}'

async def main():
    print('Start =>', now())
    print(await asyncio.gather(*[producer(f'async task {i+1}') for i in range(3)]))
    print('Stop  at', now())

asyncio.run(main())

While this code works, Python’s manuals today include a note that recommends TaskGroup over gather on the grounds of better exception handling. Should you care to follow that advice, you’ll need to move on to the next section.

Running concurrent tasks with “async with” and “async for”

As yet another concurrency option, the combination of the async with statement and a manager object can automatically handle both protocols and exceptions, and wait for tasks to finish. As half of this pair, async with internally awaits the awaitables returned by its subject’s __aenter__ and __aexit__ methods, the same way that the normal with statement calls the context-manager protocol’s __enter__ and __exit__.

Having said that, we unfortunately won’t study either the normal with or its method protocols until Part VII, because they require knowledge of classes and OOP (yet again, Python’s toolset assumes you must already know Python to use Python!) so you’ll have to take this on faith for now. In the following code, though, it’s enough to know that using a TaskGroup asynchronous context manager in concert with async with will automatically await results from all of its tasks before the statement exits, as well as handle some thorny issues involving exceptions:

async def producer(label):
    await asyncio.sleep(2)
    return f'All done, {label}, {now()}'

async def main():
    print('Start =>', now())
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(producer(f'async task {i+1}')) for i in range(3)]
    for task in tasks:
        print(task.result())
    print('Stop  at', now())

asyncio.run(main())

The net effect runs all three producer calls concurrently in two seconds as before:

Start => [19:00:44]
All done, async task 1, [19:00:46]
All done, async task 2, [19:00:46]
All done, async task 3, [19:00:46]
Stop  at [19:00:46]

Finally, when a function is coded with async def and also uses yield, it’s considered an asynchronous generator. When called, it returns an asynchronous iterable object which uses the __anext__ method noted earlier, and can be used in an async for statement to execute the body of the function. (And no, there are no async variants of other statements like while or if; with and for were the only targets deemed worthy of the aysnc twist—so far).

Internally, async for uses methods __aiter__ and __anext__ much like the normal iteration protocol’s __iter__ and __next__ that we studied in Chapter 14. The __anext__ method returns an awaitable that produces a next value, and raises StopAsyncIteration to signal the end of values, instead of StopIteration. async for simply gets a next value by awaiting a call to __anext__, and running the loop’s block of code for every value obtained this way. This encroaches on classes and OOP again, but statement users may ignore the details.

To avoid confusion, keep in mind that async for awaits the next item in the iterable it scans—which is not the same as awaiting for something else in the loop body of a normal for, as in prior examples here. Moreover, async for does not automatically parallelize anything—it’s really just like a normal for, but adds an await for each next item obtained from its iterable, instead of issuing a potentially blocking call. async for’s implicit awaits between loops allow it to be interleaved with other ready-to-run tasks, but don’t help much otherwise:

async def producer(label):
    for i in range(3):
        await asyncio.sleep(2)
        yield f'All done, {label} {i+1}, {now()}'

async def main():
    print('Start =>', now())
    async for reply in producer('async task'):
        print(reply)
    print('Stop  at', now())

asyncio.run(main())

As coded here, the result is the same six-second showing of the original serial version, because there’s nothing else to run in this simple demo during the sleeps:

Start => [19:00:46]
All done, async task 1, [19:00:48]
All done, async task 2, [19:00:50]
All done, async task 3, [19:00:52]
Stop  at [19:00:52]

And lest this section hasn’t yet managed to send you screaming into the night, it’s also possible to use await and async for within a comprehension that’s coded in an async def—but this must remain a story for another day:

async def hmm():
    result = [i async for i in asynciter() if i > 0]      # This code is not real
    result = [await func() for func in coroutines]        # It's also unreal…