The iter and next built-ins

To simplify manual iteration code, Python also provides a built-in function, next, that has the same net effect as calling an object’s __next__ method. That is, given an iterator object X, the call next(X) is the same as X.__next__(), but is noticeably simpler to type and read (and actually runs slightly faster in CPython 3.12 for tested cases). With files, for instance, either form may be used:

>>> f = open('data.txt')
>>> f.__next__()                   # Call iteration method directly
'Testing file IO\n'
>>> next(f)                        # next(f) is the same as f.__next__()
'Learning Python, 6E\n'
>>> next(f)
'Python 3.12\n'
>>> next(f)
…exception text omitted from here on…
StopIteration

Technically, there is one more piece to the iteration protocol alluded to earlier. When the for loop begins, it first obtains an iterator from an iterable object, by calling the iterable’s __iter__ method. The object returned by this call in turn has the required __next__ method to advance. For convenience again, the iter built-in function internally runs the equivalent of the __iter__ method, much as next runs the equivalent of __next__.

Hence, for loops run the internal equivalent of the following, though the iter step is moot and optional for files—they are their own iterators, because files don’t support multiple scans per open:

>>> f = open('data.txt')
>>> I = iter(f)                    # Fetch an iterator from an iterable
>>> next(I)                        # Fetch the next result form the iterator
'Testing file IO\n'
>>> next(I)                        # Files iterables are iterators themselves
'Learning Python, 6E\n'
…etc…

The full iteration protocol

With all these pieces in place, Figure 14-1 sketches this full iteration protocol, used by every iteration tool in Python and supported by a wide variety of object types. It’s based on two objects used in two distinct steps by iteration tools:

• The iterable object for which iteration is requested. Calling this object’s __iter__ returns an iterator, and is the same as calling iter.

• The iterator object returned by the iterable. Calling this object’s __next__ produces results during the iteration and raises StopIteration when no more results remain, and is the same as calling next.

Figure 14-1. The iteration protocol, used by for loops, comprehensions, maps, and more

These steps are orchestrated automatically by iteration tools in most cases, but it helps to understand these two objects’ roles. For example, in some cases these two objects are the same when only a single scan is supported (e.g., files), and the iterator object is often a temporary, used internally by the iteration tool.

Moreover, some objects are both an iteration tool (they iterate) and an iterable object (their results are iterable)—including the enumerate and zip built-ins, and Chapter 20’s generator expressions. Such iterables already avoid constructing result lists in memory themselves, but applying them to other iterables saves even more space. When such tools are combined, no work is done until an iteration tool requests results.

In code, the protocol’s first step becomes obvious if we look at how for loops internally process built-in sequence types such as lists (for uses the internal equivalents of the “__” methods, but you use either in your code):

>>> L = [1, 2]
>>> I = iter(L)                 # Obtain an iterator object from an iterable
>>> next(I)                     # Call next(iterator) to advance to next item
1
>>> next(I)                     # Or use L.__iter__() and I.__next__() calls
2
>>> next(I)
StopIteration

As we saw earlier, the initial iter step is not required for files, because a file object supports just one scan and hence is its own iterator. You can see this yourself with is (recall from Chapter 9 that this means object identity—the same exact piece of object memory, not just the same value):

>>> f = open('data.txt')
>>> iter(f) is f                # Files are iterators themselves: iter() optional
True
>>> iter(f) is f.__iter__()     # Both calls return file object f itself
True
>>> next(f)                     # Which responds to next requests directly
'Testing file IO\n'

Lists and many other built-in objects, though, are not their own iterators because they do support multiple open iterations—there may be any number of iterations active in nested loops, and all may be at different positions at a given point in time. For such objects, we must call iter to start iterating manually:

>>> L = [1, 2, 3]
>>> iter(L) is L                # Lists are not their own iterators: use iter()
False
>>> next(L)
TypeError: 'list' object is not an iterator

>>> I = iter(L)                 # Same as L.__iter__()
>>> next(I)                     # Same as I.__next__()
1

Manual iteration

Although Python iteration tools call these functions automatically, we can also use them to apply the iteration protocol manually when needed. The following demonstrates the equivalence between automatic and manual iteration (again, for runs the internal equivalent of I.__next__ instead of the next(I) used here, but the effect is the same):

>>> L = [1, 2, 3]
>>>
>>> for X in L:                   # Automatic iteration
        print(X ** 2, end=' ')    # Obtain iter, call __next__, catch exceptions

1 4 9

>>> I = iter(L)                   # Manual iteration: what for loops usually do
>>> while True:                   # Use try statements to catch stop exception
        try:
            X = next(I)
        except StopIteration:
            break
        print(X ** 2, end=' ')
   
1 4 9

To understand this code, you need to know that try statements run an action and catch exceptions that occur while the action runs (we met exceptions briefly in Chapter 10 but will explore them in depth in Part VII).

More on iter and next

For full fidelity, it should also be noted that for loops and other iteration tools can sometimes work differently for user-defined classes—repeatedly indexing an object instead of running the iteration protocol—but they prefer the iteration protocol when supported (more on this story when we study operator overloading in Chapter 30).

Though not commonly used, it’s also worth noting that next accepts an optional second default argument for an exit value; if passed, it’s returned at the end instead of raising a stop exception:

>>> L = [1]
>>> I = iter(L)                                # Result instead of exception
>>> next(I, 'end of list')
1
>>> next(I, 'end of list')
'end of list'

Combined with the := named-assignment expression, this can shave multiple lines off the preceding manual-iteration code—but will also fail if the passed default can appear as a valid result:

>>> I = iter(L)
>>> while (X := next(I, None)) != None:        # Same effect, less code
        print(X ** 2, end=' ')                 # Assuming None is safe!

Though also uncommon, iter accepts a second sentinel argument to signal stop from a callable. This very different mode provides an arguably tricky way to use for to read files by blocks—but requires info on functions or lambda, which we don’t yet have:

>>> f = open('data.txt')
>>> I = iter(lambda: f.read(5), '')            # Callables and sentinels
>>> for block in I: print(block, end='')       # Assuming you know lambda!

Watch for lambda in the next part of this book, and see Python’s docs for more on next and iter modes.

Reprise: dictionaries, range, enumerate, and zip

As we saw in the last chapter, the usual way to step through the keys of a dictionary is with a for loop:

>>> D = dict(a=1, b=2)
>>> for key in D:                # Dictionaries are implicitly iterable
        print(key, D[key])
   
a 1
b 2

This works simply because dictionaries are iterables with an iterator that automatically returns one key at a time in an iteration tool like for:

>>> I = iter(D)                  # Which just means they support the protocol
>>> next(I)
'a'
>>> next(I)
'b'
>>> next(I)
StopIteration

The iteration protocol is also the reason that we’ve had to wrap range results in a list call to see their values all at once in a REPL. Objects that are nonsequence iterables return results one at a time, not in a physical list:

>>> R = range(5)
>>> R
range(0, 5)
>>> I = iter(R)                  # Use iteration protocol to produce results
>>> next(I)
0
>>> next(I)
1
>>> list(range(5))               # Or use list() to collect all results at once
[0, 1, 2, 3, 4]

Note that the list call here is not needed and shouldn’t be used in contexts where iteration happens automatically—such as within for loops. It is needed for displaying values all at once, though, and may also be required when list-like behavior or multiple scans are required for objects that normally produce results on demand (more on this later).

Now that you have a better understanding of this protocol, you should also be able to see how it explains why the enumerate tool introduced in the prior chapter works the way it does:

>>> E = enumerate('text')        # enumerate is an iterable too
>>> E
<enumerate object at 0x1010ab880>
>>> I = iter(E)
>>> next(I)                      # Generate results with iteration protocol
(0, 't')
>>> next(I)                      # Or use list() to force generation to run
(1, 'e')
>>> list(enumerate('text'))
[(0, 't'), (1, 'e'), (2, 'x'), (3, 't')]

Unlike range, the enumerate built-in’s result is its own iterator, much like files. This means it supports just one scan per call, and iter is optional (more on this ahead too):

>>> R = range(5)
>>> iter(R) is R
False

>>> E = enumerate('text')
>>> iter(E) is E
True
>>> next(E)
(0, 't')

The zip built-in, covered in the prior chapter, work the same way in iteration tools. Like enumerate, zip is also its own iterator, so we have to call it again to iterate again:

>>> Z = zip((1, 2, 3), (10, 20, 30))
>>> Z
<zip object at 0x101236240>
>>> I = iter(Z)
>>> next(I)
(1, 10)
>>> next(I)
(2, 20)

>>> I is Z
True
>>> list(Z)
[(3, 30)]
>>> list(zip((1, 2, 3), (10, 20, 30)))
[(1, 10), (2, 20), (3, 30)]

Iterator nesting

More interestingly, zip is both iteration tool and iterable: it iterates over its arguments’ results, but also returns an iterable object with an iterator for its own results. In the following, for example, it’s a tool that receives results from two range iterables, but its own results are produced on demand as well. In other words, there are three iterables and two levels of iteration at work here:

>>> Z = zip(range(1, 4), range(10, 40))
>>> next(Z)
(1, 10)
>>> next(Z)
(2, 11)
>>> next(Z)
(3, 12)
>>> list(zip(range(1, 4), range(10, 40)))
[(1, 10), (2, 11), (3, 12)]

In fact, iterators can be nested arbitrarily. Because enumerate is both iteration tool and object too, in the following, range, enumerate, and zip all produce their results on demand, and list makes all the dances run:

>>> list(enumerate(range(1, 4)))
[(0, 1), (1, 2), (2, 3)]

>>> list(zip(enumerate(range(1, 4)), enumerate(range(5, 8))))
[((0, 1), (0, 5)), ((1, 2), (1, 6)), ((2, 3), (2, 7))]

Similarly, the following enumerates the zipped results of ranges—but only when requested by for:

>>> for x in enumerate(zip(range(1, 4), range(5, 8))): print(x)
... 
(0, (1, 5))
(1, (2, 6))
(2, (3, 7))

There are three levels of iterables in this, all deferring their results until they are activated. In practice, this works naturally, but nothing happens in such code until a tool like list or for asks for results at the top. In response, all the actors here return just one result at a time, to minimize memory requirements and avoid delays.

Functional iterables: map and filter

Like range, enumerate, and zip, the map and filter built-ins produce their results individually to conserve space and avoid pauses. Like enumerate and zip, these tools also are both iterable tools and iterable objects themselves: they scan other iterables, and produce their own results on demand.

Unlike other iterables we’ve met, though, map and filter apply function calls instead of expressions, so their complete story requires function-coding skills we won’t gain until the next part of the book. Still, we can preview their fundamentals here using built-in functions without having to code new functions of our own.

For example, the map built-in, which made a brief cameo appearance in Chapter 8 (and has nothing directly to do with mappings like dictionaries!), calls a provided function for each item in a provided iterable, and returns the collected results as another iterable. In the following, it applies the ord built-in to collect character code points:

>>> ord('p')                            # Return a single character's code point
112
>>> M = map(ord, 'py3')                 # map returns an iterable, not a list
>>> M                                   # Runs ord(x) for every x in iterable
<map object at 0x101227550>
>>> next(M)                             # Iterating manually: exhausts results
112                                     # map supports no sequence ops like [i]
>>> next(M)
121
>>> next(M)
51
>>> next(M)
StopIteration

As usual, you can force results with list if you must treat them as a list, and for automates iterations:

>>> list(map(ord, 'py3'))               # Force a real list - only if needed
[112, 121, 51]
>>> M = map(ord, 'py3')                 # Must call again to scan again
>>> for x in M: print(x, end=' ')       # Iteration tools auto call next()
112 121 51

The filter built-in, which we met momentarily in Chapter 12 and will study more fully in the next part of this book, is analogous. It returns items in an iterable for which a passed-in function returns True. In the following, we’re leveraging concepts we’ve already learned—True includes nonempty and nonzero objects, the bool built-in returns a single object’s truth value, and the str string’s isdigit method is true for all-digit text):

>>> filter(bool, ['lp6e', '', 2024])
<filter object at 0x101227850>

>>> list(filter(bool, ['lp6e', '', 2024]))           # Collect "true" items
['lp6e', 2024]
>>> list(filter(str.isdigit, ['lp6e', '2024']))      # Collect all-digit strings
['2024']

Like most of the tools discussed in this section, filter both accepts an iterable to process and returns an iterable to generate results. It doesn’t do any work until code like a for loop asks it to. As a preview, both map and filter can be emulated roughly with list comprehensions and more closely with Chapter 20’s generator expressions; but to grok the following code in full, we have to await this chapter’s presentation of comprehensions coming up soon:

>>> [ord(x) for x in 'py3']
[112, 121, 51]

>>> [x for x in ['lp6e', '2024'] if x.isdigit()]
['2024']

Multiple-pass versus single-pass iterables

We’ve noted a few times that some iterables that don’t allow multiple scans are their own iterables. Since this is a subtle difference that can impact the way you’ll use them, it’s worth a separate callout here.

In particular, the range built-in’s result, along with objects like dictionaries and lists, differ from other built-ins described in this section. They are not their own iterators (you must make one with iter when iterating manually), and they support multiple iterators (each remembers its position independently):

>>> R = range(3)                           # range allows multiple iterators
>>> next(R)
TypeError: range object is not an iterator

>>> I1 = iter(R)
>>> next(I1)
0
>>> next(I1)
1
>>> I2 = iter(R)                           # Two iterators on one range
>>> next(I2)
0
>>> next(I1)                               # I1 is at a different spot than I2
2

By contrast, the results of enumerate, zip, map, filter, as well as open for files, are their own iterators, because none of these tools support multiple active iterations on the same call result. Because of this, the iter call is optional for stepping through such objects’ results (though harmless: their iter is themselves), and we must call these tools again to start a fresh iteration from the start:

>>> Z = zip((1, 2, 3), (10, 11, 12))
>>> I1 = iter(Z)
>>> I2 = iter(Z)                           # Two iterators on one zip
>>> next(I1)
(1, 10)
>>> next(I1)
(2, 11)
>>> next(I2)                               # But I2 is at same spot as I1!
(3, 12)

>>> M = map(ord, 'py3')                    # Ditto for map (and others)
>>> I1, I2 = iter(M), iter(M)
>>> next(I1), next(I1)
(112, 121)
>>> next(I2)
51

>>> L = [0, 1, 2]                          # But lists (and others) do many scans
>>> I1, I2 = iter(L), iter(L)
>>> next(I1), next(I1)
(0, 1)
>>> next(I2)                               # Multiple active scans, like range
0

When we code our own iterable objects with classes later in the book (Chapter 30), you’ll see that multiple iterators are usually supported by returning new objects for the iter call; a single iterator generally means an object returns itself and supports next directly. In Chapter 20, you’ll also find that generator functions and expressions behave like map and zip instead of range in this regard, supporting just a single active iteration scan. Also in that chapter, you’ll see some subtle implications of single-pass iterators in loops that attempt to scan multiple times—code that treats these as lists may fail without manual list conversions.

Standard-library iterables in Python

Finally, while out of scope here, and technically part of its standard library instead of its language, Python provides additional tools that support the iteration protocol and thus may also be used in for loops other iteration tools.

For instance, shelves (an access-by-key filesystem for Python objects), as well as the results of os.popen (a tool for reading the output of shell commands), are iterables that can be processed with the full set of iteration tools. The standard directory (a.k.a. folder) walker in Python, os.walk, is iterable too, but we’ll save details and an example until Chapter 20’s coverage of this tool’s basis—generators and yield.

Ultimately, all such tools implement the iter/next interface defined by the iteration protocol. We don’t normally see this machinery because for and its kin run it for us automatically to step through results. In fact, everything that scans left to right in Python employs the iteration protocol in the same way—including the topic of the next section.

Filter clauses: if

As one particularly useful extension, the for loop nested in a comprehension expression can have an associated if clause to filter out of the result items for which the test is not true. (It’s really a filter in, but it works out the same).

For example, suppose we want to repeat the prior section’s file-scanning example, but we need to collect only lines that begin with the letters L or P (perhaps the first character on each line is an action code of some sort). Adding a simple if filter clause to our expression does the trick:

>>> lines = [line.rstrip() for line in open('data.txt') if line[0] in 'LP']
>>> lines
['Learning Python, 6E', 'Python 3.12']

Here, the if clause checks each line read from the file to see whether its first character matches; if not, the line is omitted from the result list, and the iteration continues. This is a fairly big expression, but it’s easy to understand if we translate it to its simple for loop statement equivalent:

>>> res = []
>>> for line in open('data.txt'):
        if line[0] in 'LP':
            res.append(line.rstrip())
   
>>> res
['Learning Python, 6E', 'Python 3.12']

In general, we can always translate a list comprehension to a for statement by appending as we go and further indenting each successive part. The converse isn’t true: for statements are more general and can address additional roles out of scope for comprehensions, but the latter’s results collection is a very common task.

This for statement equivalent works, but it takes up four lines instead of one and may run slower. In fact, you can squeeze a substantial amount of logic into a list comprehension when you need to—the following works like the prior but selects only lines that end in a digit (before the newline at the end), by filtering with a more sophisticated expression on the right side (which uses [-1:] instead of [-1] to handle files with blank lines empty after rstrip):

>>> [line.rstrip() for line in open('data.txt') if line.rstrip()[-1:].isdigit()]
['Python 3.12']

As another if filter example, the first result in the following gives the total lines in a text file, and the second strips whitespace on both ends to omit blank lines in the tally in just one line of code:

>>> fname = 'data-blank-lines.txt'
>>> len(open(fname).readlines())                                 # All lines
5
>>> len([line for line in open(fname) if line.strip() != ''])    # Nonblank lines
3

Nested loops: for

List comprehensions can become even more complex if we need them to—for instance, they may contain nested loops, coded as a series of for clauses. In fact, their full syntax allows for any number of for clauses, each of which can have an optional associated if clause.

For example, the following builds a list of the concatenation of x + y for every x in one string and every y in another. It effectively collects all the ordered combinations of the characters in two strings:

>>> [x + y for x in 'abc' for y in '123']
['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3']

Again, one way to understand this expression is to convert it to statement form by indenting its parts. The following is an equivalent, but likely slower, alternative way to achieve the same effect (it’s the same as the first nested for example in the prior chapter, but builds a list of results instead of simply printing them):

>>> res = []
>>> for x in 'abc':
        for y in '123':
            res.append(x + y)
   
>>> res
['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3']

Beyond this complexity level, though, list comprehension expressions can sometimes become too compact for their own good. In general, they are intended for simple types of iterations; for more involved work, a simpler for statement structure will probably be easier to understand and modify in the future. As usual in programming, if something is difficult for you to understand, it’s probably not the best idea.