The contextlib Utilities

Before rolling your own context manager classes, take a look at contextlib—“Utilities for with-statement contexts” in the Python documentation. Maybe what you are about to build already exists, or there is a class or some callable that will make your job easier.

Besides the redirect_stdout context manager mentioned right after Example 18-3, redirect_stderr was added in Python 3.5—it does the same as the former, but for output directed to stderr.

The contextlib package also includes:

closing

A function to build context managers out of objects that provide a close() method but don’t implement the __enter__/__exit__ interface.

suppress

A context manager to temporarily ignore exceptions given as arguments.

nullcontext

A context manager that does nothing, to simplify conditional logic around objects that may not implement a suitable context manager. It serves as a stand-in when conditional code before the with block may or may not provide a context manager for the with statement—added in Python 3.7.

The contextlib module provides classes and a decorator that are more widely applicable than the decorators just mentioned:

@contextmanager

A decorator that lets you build a context manager from a simple generator function, instead of creating a class and implementing the interface. See “Using @contextmanager”.

AbstractContextManager

An ABC that formalizes the context manager interface, and makes it a bit easier to create context manager classes by subclassing—added in Python 3.6.

ContextDecorator

A base class for defining class-based context managers that can also be used as function decorators, running the entire function within a managed context.

ExitStack

A context manager that lets you enter a variable number of context managers. When the with block ends, ExitStack calls the stacked context managers’ __exit__ methods in LIFO order (last entered, first exited). Use this class when you don’t know beforehand how many context managers you need to enter in your with block; for example, when opening all files from an arbitrary list of files at the same time.

With Python 3.7, contextlib added AbstractAsyncContextManager, @asynccontextmanager, and AsyncExitStack. They are similar to the equivalent utilities without the async part of the name, but designed for use with the new async with statement, covered in Chapter 21.

The most widely used of these utilities is the @contextmanager decorator, so it deserves more attention. That decorator is also interesting because it shows a use for the yield statement unrelated to iteration.

Using @contextmanager

The @contextmanager decorator is an elegant and practical tool that brings together three distinctive Python features: a function decorator, a generator, and the with statement.

Example 18-5 replaces the LookingGlass class from Example 18-3 with a generator function.

import contextlib
import sys

@contextlib.contextmanager  
def looking_glass():
    original_write = sys.stdout.write  

    def reverse_write(text):  
        original_write(text[::-1])

    sys.stdout.write = reverse_write  
    yield 'JABBERWOCKY'  
    sys.stdout.write = original_write  

Apply the contextmanager decorator.

Preserve the original sys.stdout.write method.

reverse_write can call original_write later because it is available in its closure.

Replace sys.stdout.write with reverse_write.

Yield the value that will be bound to the target variable in the as clause of the with statement. The generator pauses at this point while the body of the with executes.

When control exits the with block, execution continues after the yield; here the original sys.stdout.write is restored.

Example 18-6 shows the looking_glass function in operation.

    >>> from mirror_gen import looking_glass
    >>> with looking_glass() as what:  
    ...      print('Alice, Kitty and Snowdrop')
    ...      print(what)
    ...
    pordwonS dna yttiK ,ecilA
    YKCOWREBBAJ
    >>> what
    'JABBERWOCKY'
    >>> print('back to normal')
    back to normal

The only difference from Example 18-2 is the name of the context manager: looking_glass instead of LookingGlass.

The contextlib.contextmanager decorator wraps the function in a class that implements the __enter__ and __exit__ methods.³

The __enter__ method of that class:

1. Calls the generator function to get a generator object—let’s call it gen.

2. Calls next(gen) to drive it to the yield keyword.

3. Returns the value yielded by next(gen), to allow the user to bind it to a variable in the with/as form.

When the with block terminates, the __exit__ method:

1. Checks whether an exception was passed as exc_type; if so, gen.throw(exception) is invoked, causing the exception to be raised in the yield line inside the generator function body.

2. Otherwise, next(gen) is called, resuming the execution of the generator function body after the yield.

Example 18-5 has a flaw: if an exception is raised in the body of the with block, the Python interpreter will catch it and raise it again in the yield expression inside looking_glass. But there is no error handling there, so the looking_glass generator will terminate without ever restoring the original sys.stdout.write method, leaving the system in an invalid state.

Example 18-7 adds special handling of the ZeroDivisionError exception, making it functionally equivalent to the class-based Example 18-3.

import contextlib
import sys

@contextlib.contextmanager
def looking_glass():
    original_write = sys.stdout.write

    def reverse_write(text):
        original_write(text[::-1])

    sys.stdout.write = reverse_write
    msg = ''  
    try:
        yield 'JABBERWOCKY'
    except ZeroDivisionError:  
        msg = 'Please DO NOT divide by zero!'
    finally:
        sys.stdout.write = original_write  
        if msg:
            print(msg)  

Create a variable for a possible error message; this is the first change in relation to Example 18-5.

Handle ZeroDivisionError by setting an error message.

Undo monkey-patching of sys.stdout.write.

Display error message, if it was set.

Recall that the __exit__ method tells the interpreter that it has handled the exception by returning a truthy value; in that case, the interpreter suppresses the exception. On the other hand, if __exit__ does not explicitly return a value, the interpreter gets the usual None, and propagates the exception. With @contextmanager, the default behavior is inverted: the __exit__ method provided by the decorator assumes any exception sent into the generator is handled and should be suppressed.

A little-known feature of @contextmanager is that the generators decorated with it can also be used as decorators themselves.⁵ That happens because @contextmanager is implemented with the contextlib.ContextDecorator class.

Example 18-8 shows the looking_glass context manager from Example 18-5 used as decorator.

    >>> @looking_glass()
    ... def verse():
    ...     print('The time has come')
    ...
    >>> verse()  
    emoc sah emit ehT
    >>> print('back to normal')  
    back to normal

looking_glass does its job before and after the body of verse runs.

This confirms that the original sys.write was restored.

Contrast Example 18-8 with Example 18-6, where looking_glass is used as a context manager.

An interesting real-life example of @contextmanager outside of the standard library is Martijn Pieters’ in-place file rewriting using a context manager. Example 18-9 shows how it’s used.

import csv

with inplace(csvfilename, 'r', newline='') as (infh, outfh):
    reader = csv.reader(infh)
    writer = csv.writer(outfh)

    for row in reader:
        row += ['new', 'columns']
        writer.writerow(row)

The inplace function is a context manager that gives you two handles—infh and outfh in the example—to the same file, allowing your code to read and write to it at the same time. It’s easier to use than the standard library’s fileinput.input function (which also provides a context manager, by the way).

If you want to study Martijn’s inplace source code (listed in the post), find the yield keyword: everything before it deals with setting up the context, which entails creating a backup file, then opening and yielding references to the readable and writable file handles that will be returned by the __enter__ call. The __exit__ processing after the yield closes the file handles and restores the file from the backup if something went wrong.

This concludes our overview of the with statement and context managers. Let’s turn to match/case in the context of a complete example.

Scheme Syntax

In Scheme there is no distinction between expressions and statements, like we have in Python. Also, there are no infix operators. All expressions use prefix notation like (+ x 13) instead of x + 13. The same prefix notation is used for function calls—e.g., (gcd x 13)—and special forms—e.g., (define x 13), which we’d write as the assignment statement x = 13 in Python. The notation used by Scheme and most Lisp dialects is known as S-expression.6

Example 18-10 shows a simple example in Scheme.

(define (mod m n)
    (- m (* n (quotient m n))))

(define (gcd m n)
    (if (= n 0)
        m
        (gcd n (mod m n))))

(display (gcd 18 45))

Example 18-10 shows three Scheme expressions: two function definitions—mod and gcd—and a call to display, which will output 9, the result of (gcd 18 45). Example 18-11 is the same code in Python (shorter than an English explanation of the recursive Euclidean algorithm).

def mod(m, n):
    return m - (m // n * n)

def gcd(m, n):
    if n == 0:
        return m
    else:
        return gcd(n, mod(m, n))

print(gcd(18, 45))

In idiomatic Python, I’d use the % operator instead of reinventing mod, and it would be more efficient to use a while loop instead of recursion. But I wanted to show two function definitions, and make the examples as similar as possible, to help you read the Scheme code.

Scheme has no iterative control flow commands like while or for. Iteration is done with recursion. Note how there are no assignments in the Scheme and Python examples. Extensive use of recursion and minimal use of assignment are hallmarks of programming in a functional style.⁷

Now let’s review the code of the Python 3.10 version of lis.py. The complete source code with tests is in the 18-with-match/lispy/py3.10/ directory of the GitHub repository fluentpython/example-code-2e.

Example 18-12 shows the first lines of lis.py. The use of TypeAlias and the | type union operator require Python 3.10.

import math
import operator as op
from collections import ChainMap
from itertools import chain
from typing import Any, TypeAlias, NoReturn

Symbol: TypeAlias = str
Atom: TypeAlias = float | int | Symbol
Expression: TypeAlias = Atom | list

Symbol

Just an alias for str. In lis.py, Symbol is used for identifiers; there is no string data type with operations such as slicing, splitting, etc.8

Atom

A simple syntactic element, such as a number or a Symbol—as opposed to a composite structure made of distinct parts, like a list.

Expression

The building blocks of Scheme programs are expressions made of atoms and lists, possibly nested.

The Parser

Norvig’s parser is 36 lines of code showcasing the power of Python applied to handling the simple recursive syntax of S-expression—without string data, comments, macros, and other features of standard Scheme that make parsing more complicated (Example 18-13).

def parse(program: str) -> Expression:
    "Read a Scheme expression from a string."
    return read_from_tokens(tokenize(program))

def tokenize(s: str) -> list[str]:
    "Convert a string into a list of tokens."
    return s.replace('(', ' ( ').replace(')', ' ) ').split()

def read_from_tokens(tokens: list[str]) -> Expression:
    "Read an expression from a sequence of tokens."
    # more parsing code omitted in book listing

The main function of that group is parse, which takes an S-expression as a str and returns an Expression object, as defined in Example 18-12: an Atom or a list that may contain more atoms and nested lists.

Norvig uses a smart trick in tokenize: he adds spaces before and after each parenthesis in the input and then splits it, resulting in a list of syntactic tokens with '(' and ')' as separate tokens. This shortcut works because there is no string type in the little Scheme of lis.py, so every '(' or ')' is an expression delimiter. The recursive parsing code is in read_from_tokens, a 14-line function that you can read in the fluentpython/example-code-2e repository. I will skip it because I want to focus on the other parts of the interpreter.

Here are some doctests extracted from lispy/py3.10/examples_test.py:

>>> from lis import parse
>>> parse('1.5')
1.5
>>> parse('ni!')
'ni!'
>>> parse('(gcd 18 45)')
['gcd', 18, 45]
>>> parse('''
... (define double
...     (lambda (n)
...         (* n 2)))
... ''')
['define', 'double', ['lambda', ['n'], ['*', 'n', 2]]]

The parsing rules for this subset of Scheme are simple:

1. A token that looks like a number is parsed as a float or int.

2. Anything else that is not '(' or ')' is parsed as a Symbol—a str to be used as an identifier. This includes source text like +, set!, and make-counter that are valid identifiers in Scheme but not in Python.

3. Expressions inside '(' and ')' are recursively parsed as lists containing atoms or as nested lists that may contain atoms and more nested lists.

Using the terminology of the Python interpreter, the output of parse is an AST (Abstract Syntax Tree): a convenient representation of the Scheme program as nested lists forming a tree-like structure, where the outermost list is the trunk, inner lists are the branches, and atoms are the leaves (Figure 18-1).

Figure 18-1. A Scheme lambda expression represented as source code (concrete syntax), as a tree, and as a sequence of Python objects (abstract syntax).

The Environment class extends collections.ChainMap, adding a change method to update a value inside one of the chained dicts, which ChainMap instances hold in a list of mappings: the self.maps attribute. The change method is needed to support the Scheme (set! …) form, described later; see Example 18-14.

class Environment(ChainMap[Symbol, Any]):
    "A ChainMap that allows changing an item in-place."

    def change(self, key: Symbol, value: Any) -> None:
        "Find where key is defined and change the value there."
        for map in self.maps:
            if key in map:
                map[key] = value  # type: ignore[index]
                return
        raise KeyError(key)

Note that the change method only updates existing keys.⁹ Trying to change a key that is not found raises KeyError.

This doctest shows how Environment works:

>>> from lis import Environment
>>> inner_env = {'a': 2}
>>> outer_env = {'a': 0, 'b': 1}
>>> env = Environment(inner_env, outer_env)
>>> env['a']  
2
>>> env['a'] = 111  
>>> env['c'] = 222
>>> env
Environment({'a': 111, 'c': 222}, {'a': 0, 'b': 1})
>>> env.change('b', 333)  
>>> env
Environment({'a': 111, 'c': 222}, {'a': 0, 'b': 333})

When reading values, Environment works as ChainMap: keys are searched in the nested mappings from left to right. That’s why the value of a in the outer_env is shadowed by the value in inner_env.

Assigning with [] overwrites or inserts new items, but always in the first mapping, inner_env in this example.

env.change('b', 333) seeks the 'b' key and assigns a new value to it in-place, in the outer_env.

Next is the standard_env() function, which builds and returns an Environment loaded with predefined functions, similar to Python’s __builtins__ module that is always available (Example 18-15).

def standard_env() -> Environment:
    "An environment with some Scheme standard procedures."
    env = Environment()
    env.update(vars(math))   # sin, cos, sqrt, pi, ...
    env.update({
            '+': op.add,
            '-': op.sub,
            '*': op.mul,
            '/': op.truediv,
            # omitted here: more operator definitions
            'abs': abs,
            'append': lambda *args: list(chain(*args)),
            'apply': lambda proc, args: proc(*args),
            'begin': lambda *x: x[-1],
            'car': lambda x: x[0],
            'cdr': lambda x: x[1:],
            # omitted here: more function definitions
            'number?': lambda x: isinstance(x, (int, float)),
            'procedure?': callable,
            'round': round,
            'symbol?': lambda x: isinstance(x, Symbol),
    })
    return env

To summarize, the env mapping is loaded with:

• All functions from Python’s math module

• Selected operators from Python’s op module

• Simple but powerful functions built with Python’s lambda

• Python built-ins renamed, like callable as procedure?, or directly mapped, like round

The REPL

Norvig’s REPL (read-eval-print-loop) is easy to understand but not user-friendly (see Example 18-16). If no command-line arguments are given to lis.py, the repl() function is invoked by main()—defined at the end of the module. At the lis.py> prompt, we must enter correct and complete expressions; if we forget to close one parenthesis, lis.py crashes.¹⁰

def repl(prompt: str = 'lis.py> ') -> NoReturn:
    "A prompt-read-eval-print loop."
    global_env = Environment({}, standard_env())
    while True:
        ast = parse(input(prompt))
        val = evaluate(ast, global_env)
        if val is not None:
            print(lispstr(val))

def lispstr(exp: object) -> str:
    "Convert a Python object back into a Lisp-readable string."
    if isinstance(exp, list):
        return '(' + ' '.join(map(lispstr, exp)) + ')'
    else:
        return str(exp)

Here is a quick explanation about these two functions:

repl(prompt: str = 'lis.py> ') -> NoReturn

Calls standard_env() to provide built-in functions for the global environment, then enters an infinite loop, reading and parsing each input line, evaluating it in the global environment, and displaying the result—unless it’s None. The global_env may be modified by evaluate. For example, when a user defines a new global variable or named function, it is stored in the first mapping of the environment—the empty dict in the Environment constructor call in the first line of repl.

lispstr(exp: object) -> str

The inverse function of parse: given a Python object representing an expression, parse returns the Scheme source code for it. For example, given ['+', 2, 3], the result is '(+ 2 3)'.

The Evaluator

Now we can appreciate the beauty of Norvig’s expression evaluator—made a little prettier with match/case. The evaluate function in Example 18-17 takes an Expression built by parse and an Environment.

The body of evaluate is a single match statement with an expression exp as the subject. The case patterns express the syntax and semantics of Scheme with amazing clarity.

KEYWORDS = ['quote', 'if', 'lambda', 'define', 'set!']

def evaluate(exp: Expression, env: Environment) -> Any:
    "Evaluate an expression in an environment."
    match exp:
        case int(x) | float(x):
            return x
        case Symbol(var):
            return env[var]
        case ['quote', x]:
            return x
        case ['if', test, consequence, alternative]:
            if evaluate(test, env):
                return evaluate(consequence, env)
            else:
                return evaluate(alternative, env)
        case ['lambda', [*parms], *body] if body:
            return Procedure(parms, body, env)
        case ['define', Symbol(name), value_exp]:
            env[name] = evaluate(value_exp, env)
        case ['define', [Symbol(name), *parms], *body] if body:
            env[name] = Procedure(parms, body, env)
        case ['set!', Symbol(name), value_exp]:
            env.change(name, evaluate(value_exp, env))
        case [func_exp, *args] if func_exp not in KEYWORDS:
            proc = evaluate(func_exp, env)
            values = [evaluate(arg, env) for arg in args]
            return proc(*values)
        case _:
            raise SyntaxError(lispstr(exp))

Let’s study each case clause and what it does. In some cases I added comments showing an S-expression that would match the pattern when parsed into a Python list. Doctests extracted from examples_test.py demonstrate each case.

    case int(x) | float(x):
        return x

    case Symbol(var):
        return env[var]

    # (quote (99 bottles of beer))
    case ['quote', x]:
        return x

    # (if (< x 0) 0 x)
    case ['if', test, consequence, alternative]:
        if evaluate(test, env):
            return evaluate(consequence, env)
        else:
            return evaluate(alternative, env)

    # (lambda (a b) (/ (+ a b) 2))
    case ['lambda' [*parms], *body] if body:
        return Procedure(parms, body, env)

    # (define half (/ 1 2))
    case ['define', Symbol(name), value_exp]:
        env[name] = evaluate(value_exp, env)

    # (define (average a b) (/ (+ a b) 2))
    case ['define', [Symbol(name), *parms], *body] if body:
        env[name] = Procedure(parms, body, env)

>>> global_env = standard_env()
>>> percent = '(define (% a b) (* (/ a b) 100))'
>>> evaluate(parse(percent), global_env)
>>> global_env['%']  # doctest: +ELLIPSIS
<lis.Procedure object at 0x...>
>>> global_env['%'](170, 200)
85.0

    # (set! n (+ n 1))
    case ['set!', Symbol(name), value_exp]:
        env.change(name, evaluate(value_exp, env))

    # (gcd (* 2 105) 84)
    case [func_exp, *args] if func_exp not in KEYWORDS:
        proc = evaluate(func_exp, env)
        values = [evaluate(arg, env) for arg in args]
        return proc(*values)

    case _:
        raise SyntaxError(lispstr(exp))

>>> evaluate(parse('(lambda is not like this)'), standard_env())
Traceback (most recent call last):
    ...
SyntaxError: (lambda is not like this)

The Procedure class could very well be named Closure, because that’s what it represents: a function definition together with an environment. The function definition includes the name of the parameters and the expressions that make up the body of the function. The environment is used when the function is called to provide the values of the free variables: variables that appear in the body of the function but are not parameters, local variables, or global variables. We saw the concepts of closure and free variable in “Closures”.

We learned how to use closures in Python, but now we can dive deeper and see how a closure is implemented in lis.py:

class Procedure:
    "A user-defined Scheme procedure."

    def __init__(  
        self, parms: list[Symbol], body: list[Expression], env: Environment
    ):
        self.parms = parms  
        self.body = body
        self.env = env

    def __call__(self, *args: Expression) -> Any:  
        local_env = dict(zip(self.parms, args))  
        env = Environment(local_env, self.env)  
        for exp in self.body:  
            result = evaluate(exp, env)
        return result  

Called when a function is defined by the lambda or define forms.

Save the parameter names, body expressions, and environment for later use.

Called by proc(*values) in the last line of the case [func_exp, *args] clause.

Build local_env mapping self.parms as local variable names, and the given args as values.

Build a new combined env, putting local_env first, and then self.env—the environment that was saved when the function was defined.

Iterate over each expression in self.body, evaluating it in the combined env.

Return the result of the last expression evaluated.

There are a couple of simple functions after evaluate in lis.py: run reads a complete Scheme program and executes it, and main calls run or repl, depending on the command line—similar to what Python does. I will not describe those functions because there’s nothing new in them. My goals were to share with you the beauty of Norvig’s little interpreter, to give more insight into how closures work, and to show how match/case is a great addition to Python.

To wrap up this extended section on pattern matching, let’s formalize the concept of an OR-pattern.

A series of patterns separated by | is an OR-pattern: it succeeds if any of the subpatterns succeed. The pattern in “Evaluating numbers” is an OR-pattern:

    case int(x) | float(x):
        return x

All subpatterns in an OR-pattern must use the same variables. This restriction is necessary to ensure that the variables are available to the guard expression and the case body, regardless of the subpattern that matched.

An OR-pattern is not restricted to appear at the top level of a pattern. You can also use | in subpatterns. For example, if we wanted lis.py to accept the Greek letter λ (lambda)¹² as well as the lambda keyword, we can rewrite the pattern like this:

    # (λ (a b) (/ (+ a b) 2) )
    case ['lambda' | 'λ', [*parms], *body] if body:
        return Procedure(parms, body, env)

Now we can move to the third and last subject of this chapter: the unusual places where an else clause may appear in Python.