Item 80: Take Advantage of Each Block in try/except/else/finally

There are four distinct times when you might want to take action during exception handling in Python: in try, except, else, and finally blocks. Each block serves a unique purpose in the compound statement, and the various combinations of these blocks are useful (see Item 121: “Define a Root Exception to Insulate Callers from APIs” for another example).

def try_finally_example(filename):
    print("* Opening file")
    handle = open(filename, encoding="utf-8")  # May raise OSError
    try:
        print("* Reading data")
        return handle.read()                   # May raise UnicodeDecodeError
    finally:
        print("* Calling close()")
        handle.close()                         # Always runs after try block

filename = "random_data.txt"

with open(filename, "wb") as f:
    f.write(b"\xf1\xf2\xf3\xf4\xf5")  # Invalid utf-8

data = try_finally_example(filename)

>>>
* Opening file
* Reading data
* Calling close()
Traceback ...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in
➥position 0: invalid continuation byte

try_finally_example("does_not_exist.txt")

>>>
* Opening file
Traceback ...
FileNotFoundError: [Errno 2] No such file or directory:
➥'does_not_exist.txt'

import json

def load_json_key(data, key):
    try:
        print("* Loading JSON data")
        result_dict = json.loads(data)  # May raise ValueError
    except ValueError:
        print("* Handling ValueError")
        raise KeyError(key)
    else:
        print("* Looking up key")
        return result_dict[key]         # May raise KeyError

assert load_json_key('{"foo": "bar"}', "foo") == "bar"

>>>
* Loading JSON data
* Looking up key

load_json_key('{"foo": bad payload', "foo")

>>>
* Loading JSON data
* Handling ValueError
Traceback ...
JSONDecodeError: Expecting value: line 1 column 9 (char 8)

The above exception was the direct cause of the following
➥exception:

Traceback ...
KeyError: 'foo'

load_json_key('{"foo": "bar"}', "does not exist")

>>>
* Loading JSON data
* Looking up key
Traceback ...
KeyError: 'does not exist'

UNDEFINED = object()
def divide_json(path):
    print("* Opening file")
    handle = open(path, "r+")                       # May raise OSError
    try:
        print("* Reading data")
        data = handle.read()                        # May raise UnicodeDecodeError
        print("* Loading JSON data")
        op = json.loads(data)                       # May raise ValueError
        print("* Performing calculation")
        value = op["numerator"] / op["denominator"] # May raise ZeroDivisionError
    except ZeroDivisionError:
        print("* Handling ZeroDivisionError")
        return UNDEFINED
    else:
        print("* Writing calculation")
        op["result"] = value
        result = json.dumps(op)
        handle.seek(0)                              # May raise OSError
        handle.write(result)                        # May raise OSError
        return value
    finally:
        print("* Calling close()")
        handle.close()                              # Always runs

temp_path = "random_data.json"

with open(temp_path, "w") as f:
    f.write('{"numerator": 1, "denominator": 10}')

assert divide_json(temp_path) == 0.1

>>>
* Opening file
* Reading data
* Loading JSON data
* Performing calculation
* Writing calculation
* Calling close()

with open(temp_path, "w") as f:
    f.write('{"numerator": 1, "denominator": 0}')

assert divide_json(temp_path) is UNDEFINED

>>>
* Opening file
* Reading data
* Loading JSON data
* Performing calculation
* Handling ZeroDivisionError
* Calling close()

with open(temp_path, "w") as f:
    f.write('{"numerator": 1 bad data')

divide_json(temp_path)

>>>
* Opening file
* Reading data
* Loading JSON data
* Calling close()
Traceback ...
JSONDecodeError: Expecting ',' delimiter: line 1 column 17
➥(char 16)

with open(temp_path, "w") as f:
    f.write('{"numerator": 1, "denominator": 10}')

divide_json(temp_path)

>>>
* Opening file
* Reading data
* Loading JSON data
* Performing calculation
* Writing calculation
* Calling close()
Traceback ...
OSError: [Errno 28] No space left on device

Item 81: assert Internal Assumptions and raise Missed Expectations

Python includes the assert statement, which will raise an AssertionError exception at runtime if the given expression is a falsey value (see Item 7: “Consider Conditional Expressions for Simple Inline Logic” for background). For example, here I try to verify that two lists are not empty, and the second assertion fails because the expression is not a truthy value:

Click here to view code image

list_a = [1, 2, 3]
assert list_a, "a empty"
list_b = []
assert list_b, "b empty"  # Raises

>>>
Traceback ...
AssertionError: b empty

Python also provides the raise statement for reporting exceptional conditions to callers (see Item 32: “Prefer Raising Exceptions to Returning None” for when to use it). Here I use raise along with an if statement to report the same type of empty list problem:

Click here to view code image

class EmptyError(Exception):
    pass

list_c = []
if not list_c:
    raise EmptyError("c empty")
>>>
Traceback ...
EmptyError: c empty

The exceptions from raise statements can be caught with try/except statements (see Item 80: “Take Advantage of Each Block in try/except/else/finally”). This alternative type of control flow is the primary purpose of raise:

Click here to view code image

try:
    raise EmptyError("From raise statement")
except EmptyError as e:
    print(f"Caught: {e}")

>>>
Caught: From raise statement

But it’s also possible to catch the exceptions from assert statements at runtime:

Click here to view code image

try:
    assert False, "From assert statement"
except AssertionError as e:
    print(f"Caught: {e}")

>>>
Caught: From assert statement

Why does Python provide two different ways (raise and assert) to report exceptional situations? The reason is that they serve separate roles.

Exceptions caused by raise statements are considered part of a function’s interface, just like arguments and return values. These exceptions are meant to be caught by calling code and processed accordingly. The potential exceptions a function raises should be in the documentation (see Item 118: “Write Docstrings for Every Function, Class, and Module”) so callers know that they might need to handle them. The behavior of these raise statements should also be verified in automated tests (see Item 109: “Prefer Integration Tests over Unit Tests”).

Exceptions caused by assert statements are not meant to be caught by callers of a function. They’re used to verify assumptions in an implementation that might not be obvious to new readers of the code. Assertions are self-documenting because they evaluate the second expression (after the comma) to create a debugging error message when the condition fails. These messages can be used by error reporting and logging facilities higher in the call stack to help developers find and fix bugs (see Item 87: “Use traceback for Enhanced Exception Reporting” for an example).

Code that implements identical functionality might use assert statements, raise statements, or both, depending on the context. For example, here I define a simple class that can be used to aggregate movie ratings. It provides a robust API that validates input and reports any problems to the caller using raise:

Click here to view code image

class RatingError(Exception):
    ...

class Rating:
    def __init__(self, max_rating):
        if not (max_rating > 0):
            raise RatingError("Invalid max_rating")
        self.max_rating = max_rating
        self.ratings = []

    def rate(self, rating):
        if not (0 < rating <= self.max_rating):
            raise RatingError("Invalid rating")
        self.ratings.append(rating)

The exceptions that this class raises are meant to be caught and, presumably, reported back to the end user or API caller who sent the invalid input:

movie = Rating(5)
movie.rate(5)
movie.rate(7)  # Raises

>>>
Traceback ...
RatingError: Invalid rating

Here’s another implementation of the same functionality, but this version is not meant to report errors to the caller. Instead, this class assumes that other parts of the program have already done the necessary validation:

Click here to view code image

class RatingInternal:
    def __init__(self, max_rating):
        assert max_rating > 0, f"Invalid {max_rating=}"
        self.max_rating = max_rating
        self.ratings = []

    def rate(self, rating):
        assert 0 < rating <= self.max_rating, \    f"Invalid {rating=}"
        self.ratings.append(rating)

When an assert statement in this class raises an exception, it’s meant to report a bug in the code. The message should include information that a programmer can later use to find the cause and fix it:

Click here to view code image

movie = RatingInternal(5)
movie.rate(5)
movie.rate(7)  # Raises

>>>
Traceback ...
AssertionError: Invalid rating=7

For assertions like this to be useful, it’s critical that calling code does not catch and silence AssertionError or Exception exceptions (see Item 85: “Beware of Catching the Exception Class”).

Ultimately, it’s on you to decide whether raise or assert will be the most appropriate choice. As the complexity of a Python program grows, the layers of interconnected functions, classes, and modules begin to take shape. Some of these systems are more externally facing APIs: library functions and interfaces meant to be leveraged by other components. In those cases, raise will be most useful (see Item 121: “Define a Root Exception to Insulate Callers from APIs”). Other code is internally facing and helps one part of the program implement larger requirements. In those cases, assert is the way to go; just make sure you don’t disable assertions (see Item 90: “Never Set __debug__ to False”).

Item 82: Consider contextlib and with Statements for Reusable try/finally Behavior

The with statement in Python is used to indicate when code is running in a special context. For example, mutual-exclusion locks (see Item 69: “Use Lock to Prevent Data Races in Threads”) can be used in with statements to indicate that the indented code block runs only while the lock is held:

Click here to view code image

from threading import Lock
lock = Lock()
with lock:
    # Do something while maintaining an invariant
    ...

The example above is equivalent to this try/finally construction (see Item 80: “Take Advantage of Each Block in try/except/else/finally”) because the Lock class properly enables use in with statements:

Click here to view code image

lock.acquire()
try:
    # Do something while maintaining an invariant
    ...
finally:
    lock.release()

The with statement version of this is better because it eliminates the need to write the repetitive code of the try/finally compound statement, ensuring that you don’t forget to have a corresponding release call for every acquire call.

It’s easy to make your objects and functions work in with statements by using the contextlib built-in module. This module contains the contextmanager decorator (see Item 38: “Define Function Decorators with functools.wraps” for background), which lets a simple function be used in with statements. This is much easier than defining a new class with the special methods __enter__ and __exit__ (the standard object-oriented approach).

For example, say that I want a region of code to have more debug logging sometimes. Here I define a function that does logging at two severity levels:

Click here to view code image

import logging

def my_function():
    logging.debug("Some debug data")
    logging.error("Error log here")
    logging.debug("More debug data")

The default logging level for my program is WARNING, so only the logging.error message will print to screen when I run the function:

my_function()

>>>
Error log here

I can elevate the logging level of this function temporarily by defining a context manager. This helper function boosts the logging severity level before running the code in the with block and reduces the logging severity level afterward:

Click here to view code image

from contextlib import contextmanager

@contextmanager
def debug_logging(level):
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield
    finally:
        logger.setLevel(old_level)

The yield expression is the point at which the with block’s contents will execute (see Item 43: “Consider Generators Instead of Returning Lists” for background). Any exceptions that happen in the with block will be re-raised by the yield expression for you to catch in the helper function (see Item 47: “Manage Iterative State Transitions with a Class Instead of the Generator throw Method” for how that works).

Now I can call the same logging function again but in the debug_logging context. This time, all the debug messages are printed to the screen in the with block. The same function running outside the with block won’t print debug messages:

Click here to view code image

with debug_logging(logging.DEBUG):
    print("* Inside:")
    my_function()

print("* After:")
my_function()
>>>
* Inside:
Some debug data
Error log here
More debug data
* After:
Error log here

with open("my_output.txt", "w") as handle:
    handle.write("This is some data!")

handle = open("my_output.txt", "w")
try:
    handle.write("This is some data!")
finally:
    handle.close()

@contextmanager
def log_level(level, name):
    logger = logging.getLogger(name)
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield logger
    finally:
        logger.setLevel(old_level)

with log_level(logging.DEBUG, "my-log") as my_logger:
    my_logger.debug(f"This is a message for {my_logger.name}!")
    logging.debug("This will not print")

>>>
This is a message for my-log!

logger = logging.getLogger("my-log")
logger.debug("Debug will not print")
logger.error("Error will print")

>>>
Error will print

with log_level(logging.DEBUG, "other-log") as my_logger:# Changed
    my_logger.debug(f"This is a message for {my_logger.name}!")
    logging.debug("This will not print")

>>>
This is a message for other-log!

Item 83: Always Make try Blocks as Short as Possible

When handling an expected exception, there’s quite a bit of overhead in getting all the various statement blocks set up properly (see Item 80: “Take Advantage of Each Block in try/except/else/finally”). For example, say that I want to make a remote procedure call (RPC) via a connection, which might encounter an error:

Click here to view code image

connection = ...
class RpcError(Exception):
    ...

def lookup_request(connection):
    ...

    raise RpcError("From lookup_request")
def close_connection(connection):
    ...
    print("Connection closed")

try:
    request = lookup_request(connection)
except RpcError:
    print("Encountered error!")
    close_connection(connection)

>>>
Error!
Connection closed

Later, imagine that I want to do more processing with the data gathered inside the try block or handle special cases. The simplest and most natural way to do this is to add code right where it’s needed. Here I change the example above to have a fast path that checks for cached responses in order to avoid extra processing:

Click here to view code image

def lookup_request(connection):
    # No error raised
    ...

def is_cached(connection, request):
    ...
    raise RpcError("From is_cached")

try:
    request = lookup_request(connection)
    if is_cached(connection, request):
        request = None
except RpcError:
    print("Encountered error!")
    close_connection(connection)

>>>
Connection closed

The problem is that the is_cached function might also raise an RpcError exception. By calling lookup_request and is_cached in the same try/except statement, later on in the code I can’t tell which of these functions actually raised the error and caused the connection to be closed:

Click here to view code image

if is_closed(connection):
    # Was the connection closed because of an error
    # in lookup_request or is_cached?
    ...

Instead, what you should do is put only one source of expected errors in each try block. Everything else should either be in an associated else block or a separate subsequent try statement:

Click here to view code image

try:
    request = lookup_request(connection)
except RpcError:
    close_connection(connection)
else:
    if is_cached(connection, request):  # Moved
        request = None

>>>
Traceback ...
RpcError: From is_cached

This approach ensures that exceptions you did not expect, such as those potentially raised by is_cached, will bubble up through your call stack and produce error messages that you can find, debug, and fix later.

Item 84: Beware of Exception Variables Disappearing

Unlike for loop variables (see Item 20: “Never Use for Loop Variables After the Loop Ends”), exception variables are not accessible on the line following an except block:

Click here to view code image

try:
    raise MyError(123)
except MyError as e:
    print(f"Inside {e=}")

print(f"Outside {e=}")  # Raises

>>>
Inside e=MyError(123)
Traceback ...
NameError: name 'e' is not defined

You might assume that the exception variable will still exist within scope of the finally block that is part of the exception handling machinery (see Item 80: “Take Advantage of Each Block in try/except/else/finally”). Unfortunately, it will not:

Click here to view code image

try:
    raise MyError(123)
except MyError as e:
    print(f"Inside {e=}")
finally:
    print(f"Finally {e=}")  # Raises

>>>
Inside e=MyError(123)
Traceback ...
NameError: name 'e' is not defined

Sometimes it’s useful to save the result of each potential outcome of a try statement. For example, say that I want to log the result of each branch for debugging purposes. In order to accomplish this, I need to create a new variable and assign a value to it in each branch:

Click here to view code image

result = "Unexpected exception"
try:
    raise MyError(123)
except MyError as e:
    result = e
except OtherError as e:
    result = e
else:
    result = "Success"
finally:
    print(f"Log {result=}")

>>>
Log result=MyError(123)

It’s important to note that the result variable is assigned in the example above even before the try block. This is necessary to address the situation where an exception is raised that is not covered by one of the except clauses. If you don’t assign result up front, a runtime error will be raised instead of your original error:

Click here to view code image

try:
    raise OtherError(123)  # Not handled
except MyError as e:
    result = e
else:
    result = "Success"
finally:
    print(f"{result=}")    # Raises

>>>
Traceback ...
OtherError: 123

The above exception was the direct cause of the following
➥exception:

Traceback ...
NameError: name 'result' is not defined

This illustrates another way in which Python does not consistently scope variables to functions. The lifetime of a variable in an except block, generator expression, list comprehension, or for loop might be different than what you expect (see Item 42: “Reduce Repetition in Comprehensions with Assignment Expressions” for an example).

Item 85: Beware of Catching the Exception Class

Errors in programs—both expected and unexpected—happen frequently. These issues are an aspect of building software that programmers must accept and attempt to mitigate. For example, say that I want to analyze the sales of a pizza restaurant and generate a daily summary report. Here I accomplish this with a simple pipeline of functions:

def load_data(path):
    ...

def analyze_data(data):
    ...

def run_report(path):
    data = load_data(path)
    summary = analyze(data)
    return summary

It would be useful to have an up-to-date summary always available, so I’ll call the run_report function on a schedule every 5 minutes. However, sometimes transient errors might occur, such as at the beginning of the day before the restaurant opens, when no transactions have yet to be recorded in the daily file:

Click here to view code image

summary = run_report("pizza_data-2024-01-28.csv")
print(summary)

>>>
Traceback ...
FileNotFoundError: [Errno 2] No such file or directory:
➥'pizza_data-2024-01-28.csv'

Normally, I’d solve this problem by wrapping the run_report call with a try/except statement that logs a failure message to the console (see Item 80: “Take Advantage of Each Block in try/except/else/finally” for details):

Click here to view code image

try:
    summary = run_report("pizza_data.csv")
except FileNotFoundError:
    print("Transient file error")
else:
    print(summary)

>>>
Transient file error

This will avoid one kind of problem, but the pipeline might raise many other types of transient exceptions that I haven’t anticipated. I want to prevent any intermittent errors from crashing the rest of the restaurant’s point-of-sale program in which the report is running. It’s more important for transactions to keep being processed than for the periodic report to be refreshed.

One way to insulate the rest of the system from failures is to catch the broader Exception parent class instead of the more specific FileNotFoundError class:

Click here to view code image

try:
    summary = run_report("pizza_data.csv")
except Exception:  # Changed
    print("Transient report issue")
else:
    print(summary)

>>>
Transient report issue

When an exception is raised, each of the except clauses is considered in order. If the exception value’s type is a subclass of the clause’s specified class, then the corresponding error handling code will be executed. By providing the Exception class to match, I’ll catch errors of any kind because they all inherit from this parent class.

Unfortunately, this approach has a big problem: The try/except statement might prevent me from noticing legitimate problems with my code. Once the pizza restaurant opens and the data file is definitely present, the run_report function surprisingly still fails. The cause is a typo in the original definition of run_report that called the analyze function—which does not exist—instead of the correct analyze_data function:

Click here to view code image

run_report("my_data.csv")


>>>

Traceback ...
NameError: name 'analyze' is not defined

Due to the highly dynamic nature of Python, the interpreter will only detect that the function is missing at execution time, not when the program first loads (see Item 3: “Never Expect Python to Detect Errors at Compile Time” for details). The interpreter will raise a NameError, which is a subclass of the Exception class. Thus, the corresponding except clause will catch the exception and report it as a transient error even though it’s actually a critical problem.

One way to mitigate this issue is to always print or log the exception that’s caught when matching the Exception class. At least that way the details about the error received will be visible; anyone looking at the console output might notice that there’s a real bug in the program. For example, here I print both the exception value and its type to make it abundantly clear what went wrong:

Click here to view code image

try:
    summary = run_report("my_data.csv")
except Exception as e:
    print("Fail:", type(e), e)
else:
    print(summary)

>>>
Fail: <class 'NameError'> name 'analyze' is not defined

There are other ways that overly broad exception handling can cause problems that are worth knowing as well (see Item 86: “Understand the Difference Between Exception and BaseException” and Item 89: “Always Pass Resources into Generators and Have Callers Clean Them Up Outside”). In addition, there are more robust ways to report and handle errors for explicit APIs that help avoid these problems (see Item 121: “Define a Root Exception to Insulate Callers from APIs”). Catching exceptions to isolate errors can be useful, but you need to ensure that you’re not accidentally hiding issues.

Item 86: Understand the Difference Between Exception and BaseException

Python documentation will tell you that programmer-defined exception classes must inherit from the Exception class. But the root of the exception tree in Python is actually BaseException, which is the parent class of Exception. Branching off from BaseException are other exception classes that Python uses for its own internal purposes.

For example, when a user presses the Control-C key combination while a Python program runs, they expect to interrupt the running program and cause it to terminate. The precise way Python accomplishes this is platform dependent, but ultimately the interpreter runtime converts the interrupt signal into a KeyboardInterrupt exception and raises it in the program’s main thread. KeyboardInterrupt does not inherit from Exception, which means that it should bypass exception handlers all the way up to the entry point of the program and cause it to exit with an error message. Here I show this behavior in action by exiting an infinite loop even though it catches the Exception class:

Click here to view code image

def do_processing():
    ...

def main(argv):
    while True:
        try:
            do_processing()  # Interrupted
        except Exception as e:
            print("Error:", type(e), e)

    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))

>>>
Traceback ...
KeyboardInterrupt

Knowing this is possible, I might choose to catch the BaseException class so I can always do cleanup before program termination, such as flushing open files to disk to ensure that they’re not corrupted. In other situations, catching a broad class of exceptions like this can be useful for insulating components against potential errors and providing resilient APIs (see Item 85: “Beware of Catching the Exception Class” and Item 121: “Define a Root Exception to Insulate Callers from APIs”). I can return 1 at the end of the exception handler to indicate that the program should exit with an error code:

Click here to view code image

def do_processing(handle):
    ...

def main(argv):
    data_path = argv[1]
    handle = open(data_path, "w+")

    while True:
        try:
            do_processing(handle)
        except Exception as e:
            print("Error:", type(e), e)
        except BaseException:
            print("Cleaning up interrupt")
            handle.flush()
            handle.close()
            return 1

    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))

>>>
Cleaning up interrupt
Traceback ...
SystemExit: 1

The problem is that there are other exception types that also inherit from BaseException, including SystemExit (caused by the sys.exit built-in function) and GeneratorExit (see Item 89: “Always Pass Resources into Generators and Have Callers Clean Them Up Outside”). Python might add more in the future as well. Python treats these exceptions as mechanisms for executing desired behavior instead of reporting error conditions, which is why they’re in a separate part of the class hierarchy. The runtime relies on users generally not catching these exceptions in order to work properly; if you catch them, you might inadvertently cause harmful side effects in a program.

Thus, if you want to achieve this type of cleanup behavior, it’s better to use constructs like try/finally statements (see Item 80: “Take Advantage of Each Block in try/except/else/finally”) and with statements (see Item 82: “Consider contextlib and with Statements for Reusable try/finally Behavior”). These constructs will ensure that the cleanup methods run regardless of whether the exception raised inherits from Exception or BaseException:

Click here to view code image

def main(argv):
    data_path = argv[1]
    handle = open(data_path, "w+")

    try:
        while True:
            try:
                do_processing(handle)
            except Exception as e:
                print("Error:", type(e), e)

    finally:
        print("Cleaning up finally")  # Always runs
        handle.flush()
        handle.close()

if __name__ == "__main__":
    sys.exit(main(sys.argv))

>>>
Cleaning up finally
Traceback ...
KeyboardInterrupt

If for some reason you really must catch and handle a direct child class of BaseException, it’s important to propagate the error correctly so other code higher up in the call stack will still receive it. For example, I might catch KeyboardInterrupt exceptions and ask the user to confirm their intention to terminate the program. Here I use a bare raise at the end of the exception handler to ensure that the exception continues normally, without modifications to its traceback (see Item 87: “Use traceback for Enhanced Exception Reporting” for background):

Click here to view code image

def main(argv):
    while True:
        try:
            do_processing()
        except Exception as e:
            print("Error:", type(e), e)
        except KeyboardInterrupt:
            found = input("Terminate? [y/n]: ")
            if found == "y":
                raise  # Propagate the error
if __name__ == "__main__":
    sys.exit(main(sys.argv))

>>>
Terminate? [y/n]: y
Traceback ...
KeyboardInterrupt

Another situation where you might decide to catch BaseException is for enhanced logging utilities (see Item 87: “Use traceback for Enhanced Exception Reporting” for a related use case). For example, I can define a function decorator that logs all inputs and outputs, including raised Exception subclass values (see Item 38: “Define Function Decorators with functools.wraps” for background):

Click here to view code image

import functools

def log(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        try:
            result = func(*args, **kwargs)
        except Exception as e:
            result = e
            raise
        finally:
            print(
                f"Called {func.__name__}"
                f"(*{args!r}, **{kwargs!r}) "
                f"got {result!r}"
            )
    return wrapper

Calling a function decorated with log will print everything as expected:

Click here to view code image

@log
def my_func(x):
    x / 0

my_func(123)

>>>
Called my_func(*(123,), **{}) got ZeroDivisionError
(➥'division by zero')

Traceback ...
ZeroDivisionError: division by zero

However, if the exception that’s raised inherits from BaseException instead of Exception, the decorator will break and cause unexpected errors:

Click here to view code image

@log
def other_func(x):
    if x > 0:
        sys.exit(1)

other_func(456)

>>>
Traceback ...
SystemExit: 1

The above exception was the direct cause of the following
➥exception:

Traceback ...
UnboundLocalError: cannot access local variable 'result'
➥where it is not associated with a value

It might seem counterintuitive, but the finally clause will run even in cases where there are no except clauses present or none of the provided except clauses actually match the exception value that was raised (see Item 84: “Beware of Exception Variables Disappearing” for another example). In the case above, that’s exactly what happened: SystemExit is not a subclass of Exception, and so that handler never ran, and result was not assigned before the call to print in the finally clause. Simply catching BaseException instead of Exception solves the problem:

Click here to view code image

def fixed_log(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        try:
            result = func(*args, **kwargs)
        except BaseException as e:  # Fixed
            result = e
            raise
        finally:
            print(
                f"Called {func.__name__}"
                f"(*{args!r}, **{kwargs!r}) "
                f"got {result!r}"
            )
    return wrapper

Now the decorator works as expected for SystemExit:

Click here to view code image

@fixed_log
def other_func(x):
    if x > 0:
        sys.exit(1)

other_func(456)

>>>
Called other_func(*(456,), **{}) got SystemExit(1)
Traceback ...
SystemExit: 1

Handling BaseException and related classes can be useful, but it’s also quite tricky, so it’s important to pay close attention to the details and be careful.

Item 87: Use traceback for Enhanced Exception Reporting

When a Python program encounters a problem, an exception is often raised. If the exception is not caught and handled (see Item 80: “Take Advantage of Each Block in try/except/else/finally”), it will propagate all the way up to the entry point of the program and cause it to exit with an error code. The Python interpreter will also print out a nicely formatted stack trace or traceback to aid developers in figuring out what went wrong. For example, here I use the assert statement to cause an exception and print out the corresponding traceback (see Item 81: “assert Internal Assumptions, raise Missed Expectations” for background):

Click here to view code image

def inner_func(message):
    assert False, message

def outer_func(message):
    inner_func(message)

outer_func("Oops!")

>>>
Traceback (most recent call last):
  File "my_code.py", line 7, in <module>
    outer_func("Oops!")
    ~~~~~~~~~~^^^^^^^^^
  File "my_code.py", line 5, in outer_func
    inner_func(message)
    ~~~~~~~~~~^^^^^^^^^
  File "my_code.py", line 2, in inner_func
    assert False, message
           ^^^^^
AssertionError: Oops!

This default printing behavior can be helpful for single-threaded code where all exceptions are happening in the main thread. But it won’t work for programs or servers that are handling many requests concurrently (see Item 71: “Know How to Recognize When Concurrency Is Necessary”). If you allow exceptions from one request to propagate all the way up to the entry point, the program will crash and cause all other requests to fail as well.

One way to deal with this is to surround the root of a request handler with a blanket try statement (see Item 85: “Beware of Catching the Exception Class” and Item 86: “Understand the Difference Between Exception and BaseException”). For example, here I define a hypothetical Request class and handler function. When an exception is hit, I can print it out to the console log for the developer to see and then return an error code to the client. This ensures that all other handlers keep processing, even in the presence of bad requests:

Click here to view code image

class Request:
    def __init__(self, body):
        self.body = body
        self.response = None

def do_work(data):
    assert False, data
    ...

def handle(request):
    try:
        do_work(request.body)
    except BaseException as e:
        print(repr(e))
        request.response = 400  # Bad request error

request = Request("My message")
handle(request)

>>>
AssertionError('My message')

The problem with this code is that the string representation of the exception value doesn’t provide enough information to debug the issue. I don’t get a traceback as I would from an unhandled exception in the Python interpreter’s main thread. Fortunately, Python can fill this gap with the traceback built-in module, which allows you to extract the traceback information from an exception at runtime. Here I use the print_tb function in the traceback built-in module to print a stack trace:

Click here to view code image

import traceback
def handle2(request):
    try:
        do_work(request.body)
    except BaseException as e:
        traceback.print_tb(e.__traceback__)  # Changed
        print(repr(e))
        request.response = 400

request = Request("My message 2")
handle2(request)

>>>
  File "my_code.py", line 70, in handle2
    do_work(request.body)
    ~~~~~~~^^^^^^^^^^^^^^
  File "my_code.py", line 42, in do_work
    assert False, data
           ^^^^^
AssertionError('My message 2')

In addition to printing, you can process the traceback’s detailed information—including filename, line number, source code line, and containing function name—however you like (e.g., to display in a GUI). Here I extract the function name for each frame in the traceback and print them to the console:

Click here to view code image

def handle3(request):
    try:
        do_work(request.body)
    except BaseException as e:
        stack = traceback.extract_tb(e.__traceback__)
        for frame in stack:
            print(frame.name)
        print(repr(e))
        request.response = 400

request = Request("My message 3")
handle3(request)

>>>
handle3
do_work
AssertionError('My message 3')

Beyond printing to the console, I can also use the traceback module to provide more advanced error handling behaviors. For example, imagine that I wanted to save a log of exceptions encountered in a separate file, encoded as one JSON payload per line. Here I accomplish this with a wrapper function that processes the traceback frames:

Click here to view code image

import json

def log_if_error(file_path, target, *args, **kwargs):
    try:
        target(*args, **kwargs)
    except BaseException as e:
        stack = traceback.extract_tb(e.__traceback__)
        stack_without_wrapper = stack[1:]
        trace_dict = dict(
            stack=[item.name for item in stack_without_wrapper],
            error_type=type(e).__name__,
            error_message=str(e),
        )
        json_data = json.dumps(trace_dict)

        with open(file_path, "a") as f:
            f.write(json_data)
            f.write("\n")

Calling the wrapper with the erroring do_work function will properly encode errors and write them to disk:

Click here to view code image

log_if_error("my_log.jsonl", do_work, "First error")
log_if_error("my_log.jsonl", do_work, "Second error")
with open("my_log.jsonl") as f:
    for line in f:
        print(line, end="")

>>>
{"stack": ["do_work"], "error_type": "AssertionError",
➥"error_message": "First error"}
{"stack": ["do_work"], "error_type": "AssertionError",
➥"error_message": "Second error"}

The traceback built-in module also includes a variety of other functions that make it easy to format, print, and traverse exception stack traces in most of the ways you’ll ever need (see https://docs.python.org/3/library/traceback.html). However, you’ll still need to handle some of the edge cases yourself (see Item 88: “Consider Explicitly Chaining Exceptions to Clarify Tracebacks” for one such case).

Item 88: Consider Explicitly Chaining Exceptions to Clarify Tracebacks

Python programs raise exceptions when they encounter errors or certain conditions in code (see Item 32: “Prefer Raising Exceptions to Returning None”). For example, here I try to access a nonexistent key in a dictionary, which results in an exception being raised:

Click here to view code image

my_dict = {}
my_dict["does_not_exist"]

>>>
Traceback ...
KeyError: 'does_not_exist'

I can catch this exception and handle it by using the try statement (see Item 80: “Take Advantage of Each Block in try/except/else/finally”):

Click here to view code image

my_dict = {}
try:
    my_dict["does_not_exist"]
except KeyError:
    print("Could not find key!")

>>>
Could not find key!

If another exception is raised while I’m already handling one, the output looks quite different. For example, here I raise a newly defined MissingError exception while handling the KeyError exception:

Click here to view code image

class MissingError(Exception):
    ...
try:
    my_dict["does_not_exist"]    # Raises first exception
except KeyError:
    raise MissingError("Oops!")  # Raises second exception

>>>
Traceback ...
KeyError: 'does_not_exist'

The above exception was the direct cause of the following
➥exception:

Traceback ...
MissingError: Oops!

The MissingError exception that’s raised in the except KeyError block is the one propagated up to the caller. However, you can also see that the stack trace printed by Python included information about the exception that caused the initial problem: the KeyError raised by the my_dict["does_not_exist"] expression. This extra data is available because Python will automatically assign an exception’s __context__ attribute to the exception instance being handled by the surrounding except block. Here is the same code as above, but now I catch the MissingError exception and print its __context__ attribute to show how the exceptions are chained together:

Click here to view code image

try:
    try:
        my_dict["does_not_exist"]
    except KeyError:
        raise MissingError("Oops!")
except MissingError as e:
    print("Second:", repr(e))
    print("First: ", repr(e.__context__))

>>>
Second: MissingError('Oops!')
First:  KeyError('does_not_exist')

In complex code with many layers of error handling, it can be useful to control these chains of exceptions to make the error messages more clear. To accomplish this, Python allows for explicitly chaining together exceptions by using the from clause in the raise statement.

For example, say that I want to define a helper function that implements the same dictionary lookup behavior as above:

Click here to view code image

def lookup(my_key):
    try:
        return my_dict[my_key]
    except KeyError:
        raise MissingError

When I look up a key that is present, I retrieve the value without a problem:

my_dict["my key 1"] = 123
print(lookup("my key 1"))

>>>
123

When the given key is missing, a MissingError exception is raised as expected:

Click here to view code image

print(lookup("my key 2"))

>>>
Traceback ...
KeyError: 'my key 2'

The above exception was the direct cause of the following
➥exception:

Traceback ...
MissingError

Now imagine that I want to augment the lookup function and be able to contact a remote database server and populate the my_dict dictionary in the event that a key is missing. Here I implement this behavior, assuming contact_server will do the database communication:

Click here to view code image

def contact_server(my_key):
    print(f"Looking up {my_key!r} in server")
    ...

def lookup(my_key):
    try:
        return my_dict[my_key]
    except KeyError:
        result = contact_server(my_key)
        my_dict[my_key] = result  # Fill the local cache
        return result

Calling this function repeatedly, I can see that the server is only contacted on the first call, when the key is not yet present in the my_dict cache. The subsequent call avoids calling contact_server and returns the value that is already present in my_dict:

Click here to view code image

print("Call 1")
print("Result:", lookup("my key 2"))
print("Call 2")
print("Result:", lookup("my key 2"))

>>>
Call 1
Looking up 'my key 2' in server
Result: my value 2
Call 2
Result: my value 2

Imagine that it’s possible for the database server not to have a requested record. In this situation, perhaps the contact_server function raises a new type of exception to indicate the condition:

Click here to view code image

class ServerMissingKeyError(Exception):
    ...

def contact_server(my_key):
    print(f"Looking up {my_key!r} in server")
    ...
    raise ServerMissingKeyError
    ...

Now when I try to look up a record that’s missing, I see a traceback that includes the ServerMissingKeyError exception and the original KeyError for the my_dict cache miss:

Click here to view code image

print(lookup("my key 3"))

>>>
Looking up 'my key 3' in server
Traceback ...
KeyError: 'my key 3'

The above exception was the direct cause of the following
➥exception:

Traceback ...
ServerMissingKeyError

The problem is that the lookup function no longer adheres to the same interface as before when there was no call to contact_server. To abstract the details of the exception from the caller, I want a cache miss to always result in a MissingError, not a ServerMissingKeyError, which might be defined in a separate module that I don’t control (see Item 121: “Define a Root Exception to Insulate Callers from APIs” for exception classes in APIs).

To fix this, I can wrap the call to contact_server in another try statement, catch the ServerMissingKeyError exception, and raise a MissingError instead (matching my desired API for lookup):

Click here to view code image

def lookup(my_key):
    try:
        return my_dict[my_key]
    except KeyError:
        try:
            result = contact_server(my_key)
        except ServerMissingKeyError:
            raise MissingError       # Convert the server error
        else:
            my_dict[my_key] = result  # Fill the local cache
            return result

Trying the new implementation of lookup, I can verify that the MissingError exception is what’s propagated up to callers:

Click here to view code image

print(lookup("my key 4"))

>>>
Looking up 'my key 4' in server
Traceback ...
KeyError: 'my key 4'

The above exception was the direct cause of the following
➥exception:

Traceback ...
ServerMissingKeyError

The above exception was the direct cause of the following
➥exception:

Traceback ...
MissingError

This chain of exceptions shows three different errors because of how the except clauses are nested in the lookup function. First, the KeyError exception is raised. Then, while handling it, the contact_server function raises a ServerMissingKeyError, which is implicitly chained from the KeyError using the __context__ attribute. The ServerMissingKeyError is then caught, and the MissingError is raised with the __context__ attribute implicitly assigned to the ServerMissingKeyError currently being handled.

A lot of information was printed for this MissingError—so much that it seems like it could be confusing to programmers trying to debug a real problem. One way to reduce the output is to use the from clause in the raise statement to explicitly indicate the source of an exception. Here I hide the ServerMissingKeyError source error by having the exception handler explicitly chain the MissingError from the KeyError:

Click here to view code image

def lookup_explicit(my_key):
    try:
        return my_dict[my_key]
    except KeyError as e:              # Changed
        try:
            result = contact_server(my_key)
        except ServerMissingKeyError:
            raise MissingError from e  # Changed
        else:
            my_dict[my_key] = result
            return result

Calling the function again, I can confirm that the ServerMissingKeyError exception is no longer printed:

Click here to view code image

print(lookup_explicit("my key 5"))

>>>
Looking up 'my key 5' in server

Traceback ...
KeyError: 'my key 5'

The above exception was the direct cause of the following
➥exception:

Traceback ...
MissingError

Although in the exception output it appears that the ServerMissingKeyError is no longer associated with the MissingError exception, it is in fact still there, assigned to the __context__ attribute as before. The reason it’s not printed is that using the from e clause in the raise statement assigns the raised exception’s __cause__ attribute to the KeyError and the __suppress_context__ attribute to True. Here I show the value of these attributes to clarify what Python uses to control the printing of unhandled exceptions:

Click here to view code image

try:
    lookup_explicit("my key 6")
except Exception as e:
    print("Exception:", repr(e))
    print("Context:  ", repr(e.__context__))
    print("Cause:    ", repr(e.__cause__))
    print("Suppress: ", repr(e.__suppress_context__))

>>>
Looking up 'my key 6' in server
Exception: MissingError()
Context:   ServerMissingKeyError()
Cause:     KeyError('my key 6')
Suppress:  True

The exception chain traversal behavior that inspects __cause__ and __suppress_context__ is only present for Python’s built-in exception printer. If you use the traceback module to process Exception stack traces yourself (see Item 87: “Use traceback for Enhanced Exception Reporting”), you might notice that chained exception data seems to be missing:

Click here to view code image

import traceback

try:
    lookup("my key 7")
except Exception as e:
    stack = traceback.extract_tb(e.__traceback__)

    for frame in stack:
        print(frame.line)

>>>
Looking up 'my key 7' in server
lookup('my key 7')
raise MissingError        # Convert the server error

In order to extract the same chained exception information that Python prints for unhandled exceptions, you need to properly consider each exception’s __cause__ and __context__ attributes:

Click here to view code image

def get_cause(exc):
    if exc.__cause__ is not None:
        return exc.__cause__
    elif not exc.__suppress_context__:
        return exc.__context__
    else:
        return None

The get_cause function can be applied in a loop or recursively to construct the full stack of chained exceptions:

Click here to view code image

try:
    lookup("my key 8")
except Exception as e:
    while e is not None:
        stack = traceback.extract_tb(e.__traceback__)
        for i, frame in enumerate(stack, 1):
            print(i, frame.line)
        e = get_cause(e)
        if e:
            print("Caused by")

>>>
Looking up 'my key 8' in server
1 lookup('my key 8')
2 raise MissingError        # Convert the server error
Caused by
1 result = contact_server(my_key)
2 raise ServerMissingKeyError
Caused by
1 return my_dict[my_key]

Another alternative way to shorten the MissingError exception chain is to suppress the KeyError source for the ServerMissingKeyError raised in contact_server. Here I do this by using the from None clause in the corresponding raise statement:

Click here to view code image

def contact_server(key):
    ...
    raise ServerMissingKeyError from None  # Suppress
    ...

Calling the lookup function again, I can confirm that the KeyError is no longer in Python’s default exception handling output:

Click here to view code image

print(lookup("my key 9"))

>>>
Traceback ...

ServerMissingKeyError

The above exception was the direct cause of the following
➥exception:

Traceback ...
MissingError

Item 89: Always Pass Resources into Generators and Have Callers Clean Them Up Outside

Python provides a variety of tools, such as exception handlers (see Item 80: “Take Advantage of Each Block in try/except/else/finally”) and with statements (see Item 82: “Consider contextlib and with Statements for Reusable try/finally Behavior”), to help you ensure that resources like files, mutexes, and sockets are properly cleaned up at the right time. For example, in a normal function, a simple finally clause will be executed before the return value is actually received by the caller, making it an ideal location to reliably close file handles:

Click here to view code image

def my_func():
    try:
        return 123
    finally:
        print("Finally my_func")

print("Before")
print(my_func())
print("After")

>>>
Before
Finally my_func
123
After

In contrast, when using a generator function (see Item 43: “Consider Generators Instead of Returning Lists”), the finally clause won’t execute until the StopIteration exception is raised to indicate that the sequence of values has been exhausted (see Item 21: “Be Defensive when Iterating over Arguments”). That means the finally clause is executed after the last item is received by the caller, unlike with a normal function:

Click here to view code image

def my_generator():
    try:
        yield 10
        yield 20
        yield 30
    finally:
        print("Finally my_generator")

print("Before")
for i in my_generator():
    print(i)

print("After")

>>>
Before
10
20
30
Finally my_generator
After

However, it’s also possible for Python generators to not finish being iterated. In theory, this could prevent the StopIteration exception from ever being raised and thereby prevent the execution of the finally clause. Here I simulate this behavior by manually stepping forward the generator function’s iterator; note how "Finally my_generator" doesn’t print:

it = my_generator()
print("Before")
print(next(it))
print(next(it))
print("After")

>>>
Before
10
20
After

The finally clause hasn’t executed yet. When will it? The answer is that it depends: It might never run. If the last reference to the iterator is dropped, garbage collection is enabled, and a collection cycle runs, which should cause the generator’s finally clause to execute:

import gc
del it
gc.collect()

>>>
Finally my_generator

The mechanism that powers this is the GeneratorExit exception, which inherits from BaseException (see Item 86: “Understand the Difference Between Exception and BaseException”). Upon garbage collection, Python will send this special type of exception into the generator if it’s not exhausted (see Item 46: “Pass Iterators into Generators as Arguments Instead of Calling the send Method” for background). Normally this causes the generator to return and clear its stack, but technically you can catch this type of exception and handle it:

Click here to view code image

def catching_generator():
    try:
        yield 40
        yield 50
        yield 60
    except BaseException as e:  # Catches GeneratorExit
        print("Catching handler", type(e), e)
        raise

At the end of the exception handler, I use a bare raise keyword with no arguments to ensure that the GeneratorExit exception propagates and none of Python’s runtime machinery breaks. Here I step forward this new generator and then cause another garbage collecting cycle:

Click here to view code image

it = catching_generator()
print("Before")
print(next(it))
print(next(it))
print("After")
del it
gc.collect()

>>>
Before
40
50
After
Catching handler <class 'GeneratorExit'>

The exception handler is run separately by the gc module, not in the original call stack that created the generator and stepped it forward. What happens if a different exception is raised while handling the GeneratorExit exception? Here I define another generator to demonstrate this possibility:

Click here to view code image

def broken_generator():
    try:
        yield 70
        yield 80
    except BaseException as e:
        print("Broken handler", type(e), e)
        raise RuntimeError("Broken")

it = broken_generator()
print("Before")
print(next(it))
print("After")
del it
gc.collect()
print("Still going")

>>>
Before
70
After
Exception ignored in: <generator object broken_generator at
➥ 0x10099b2e0>
Traceback ...
RuntimeError: Broken
Broken handler <class 'GeneratorExit'>
Still going

This outcome is surprising: The gc module catches the RuntimeError raised by broken_generator and prints it out to sys.stderr. The exception is not raised back into the main thread where gc.collect was called. Instead, it’s completely swallowed and hidden from the rest of the program, which continues running. This means that you can’t rely on exception handlers or finally clauses in generators to always execute and report errors back to callers.

To work around this potential risk, you can allocate resources that need to be cleaned up outside a generator and pass them in as arguments. For example, imagine that I’m trying to build a simple utility that finds the maximum length of the first five lines of a file. Here I define a simple generator that yields line lengths given a file path:

Click here to view code image

def lengths_path(path):
    try:
        with open(path) as handle:
            for i, line in enumerate(handle):
                print(f"Line {i}")
                yield len(line.strip())
    finally:
        print("Finally lengths_path")

I can use the generator in a loop to calculate the maximum and then terminate the loop early, leaving the lengths_path generator in a partially executed state:

Click here to view code image

max_head = 0
it = lengths_path("my_file.txt")

for i, length in enumerate(it):
    if i == 5:
        break
    else:
        max_head = max(max_head, length)

print(max_head)

>>>
Line 0
Line 1
Line 2
Line 3
Line 4
Line 5
99

After the generator iterator goes out of scope sometime later, it will be garbage collected, and the finally clause will run as expected:

del it
gc.collect()

>>>
Finally lengths_path

This delayed behavior is what I’m trying to avoid. I need finally to run within the call stack of the original loop so that if any errors are encountered, they’re properly raised back to the caller. This is especially important for resources like mutex locks that must avoid deadlocking. To accomplish the correct behavior, I can pass an open file handle into the generator function:

Click here to view code image

def lengths_handle(handle):
    try:
        for i, line in enumerate(handle):
            print(f"Line {i}")
            yield len(line.strip())
    finally:
        print("Finally lengths_handle")

I can use a with statement around the loop to make sure the file is opened and closed reliably and immediately so the generator doesn’t have to manage the file handle itself:

Click here to view code image

max_head = 0

with open("my_file.txt") as handle:
    it = lengths_handle(handle)
    for i, length in enumerate(it):
        if i == 5:
            break
        else:
            max_head = max(max_head, length)

print(max_head)
print("Handle closed:", handle.closed)

>>>
Line 0
Line 1
Line 2
Line 3
Line 4
Line 5
99
Handle closed: True

Again, because the loop iteration ended before exhaustion, the generator function hasn’t exited, and the finally clause hasn’t executed. But this is okay with this different approach because I’m not relying on the generator to do any important cleanup.

The GeneratorExit exception represents a compromise between correctness and system health. If generators weren’t forced to exit eventually, all prematurely stopped generators would leak memory and potentially cause the program to crash. Swallowing errors is the trade-off that Python makes because it’s a reasonable thing to do most of the time. But it’s up to you to make sure your generators expect this behavior and plan accordingly.

Item 90: Never Set __debug__ to False

When you add an assert statement like this into a Python program:

Click here to view code image

n = 3
assert n % 2 == 0, f"{n=} not even"

it’s essentially equivalent to the following code:

Click here to view code image

if __debug__:
    if not (n % 2 == 0):
        raise AssertionError(f"{n=} not even")

>>>
Traceback ...
AssertionError: n=3 not even

You can also use the __debug__ global built-in variable directly in order to gate the execution of more complex verification code:

Click here to view code image

def expensive_check(x):
    ...

items = [1, 2, 3]
if __debug__:
    for i in items:
        assert expensive_check(i), f"Failed {i=}"

>>>
Traceback ...
AssertionError: Failed i=2

The only way to set the __debug__ built-in global variable to False is by specifying the -O command-line argument at Python startup time. For example, here is a Python invocation that will start with __debug__ equal to True (the default):

Click here to view code image

$ python3 -c 'assert False, "FAIL"; print("OK")'
Traceback ...
AssertionError: FAIL

Adding the -O command-line option causes the assert statement to be skipped entirely, resulting in different output:

Click here to view code image

$ python3 -O -c 'assert False, "FAIL"; print("OK")'
OK

Although Python is an extremely dynamic language (see Item 3: “Never Expect Python to Detect Errors at Compile Time”), it won’t allow you to modify the value of __debug__ at runtime:

Click here to view code image

__debug__ = False

>>>
Traceback ...
SyntaxError: cannot assign to __debug__

If the __debug__ constant is True, it will always stay that way during the life of a program.

The original intention of the __debug__ flag was to allow users to optimize the performance of their code by skipping seemingly unnecessary assertions at runtime. However, as time has gone on, more and more code, especially common frameworks and libraries, has become dependent on assertions being active in order to verify assumptions at program startup time and runtime. By disabling the assert statement and other debug code, you’re undermining your program’s validity for little practical gain.

If performance is what you’re after, there are far better approaches to making programs faster (see Item 92: “Profile Before Optimizing” and Item 94: “Know When and How to Replace Python with Another Programming Language”). If you have very expensive verification code that you need to disable at runtime, then create your own enable_debug helper function and associated global variables to control these debugging operations in your own code instead of relying on __debug__.

There’s still value in always keeping assertions active, especially in low-level code, even when you need to squeeze every ounce of performance out of your code, like when using MicroPython for microcontrollers (https://micropython.org). Somewhat counterintuitively, the presence of assert statements can help you debug even when these statements aren’t failing. When you get a bug report, you can use successfully passing assertions to rule out possibilities and narrow the scope of what’s gone wrong.

Ultimately, assertions are a powerful tool for ensuring correctness, and they should be used liberally throughout your code to help make assumptions explicit.

Item 91: Avoid exec and eval Unless You’re Building a Developer Tool

Python is a dynamic language that lets you do nearly anything at runtime (which can cause problems; see Item 3: “Never Expect Python to Detect Errors at Compile Time”). Many of its features enable these extremely flexible capabilities, such as setattr/getattr/hasattr (see Item 61: “Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes”), metaclasses (see Item 64: “Annotate Class Attributes with __set_name__”) and descriptors (see Item 60: “Use Descriptors for Reusable @property Methods”).

However, the most dynamic capability of all is executing arbitrary code from a string at runtime. This is possible in Python with the eval and exec built-in functions.

eval takes a single expression as a string and returns the result of its evaluation as a normal Python object:

x = eval("1 + 2")
print(x)

>>>
3

Passing a statement to eval will result in an error:

Click here to view code image

eval(
    """
if True:
    print('okay')
else:
    print('no')
""""""
)

>>>
Traceback ...
SyntaxError: invalid syntax (<string>, line 2)

Instead, you can use exec to dynamically evaluate larger chunks of Python code. exec always returns None, and to get data out of it, you need to use the global and local scope dictionary arguments. Here, when I access the my_condition variable, it bubbles up to the global scope to be resolved, and my assignment of the x variable is made in the local scope (see Item 33: “Know How Closures Interact with Variable Scope and nonlocal” for background):

Click here to view code image

global_scope = {"my_condition": False}
local_scope = {}
exec(     """
if my_condition:
    x = 'yes'
else:
    x = 'no'
"""""",
    global_scope,
    local_scope,
)
print(local_scope)

>>>
{'x': 'no'}

If you discover eval or exec in an otherwise normal application codebase, it’s often a red flag indicating that something is seriously wrong. These features can cause severe security issues if they are inadvertently connected to an input channel that gives access to an attacker. Even for plug-in architectures, where these features might seem like a natural fit, Python has better ways to achieve similar outcomes (see Item 98: “Lazy-Load Modules with Dynamic Imports to Reduce Startup Time”).

The only time it’s actually appropriate to use eval and exec is in code that supports your application with an improved development experience, such as a debugger, notebook system, run-eval-print-loop (REPL), performance benchmarking tool, code generation utility, and so on. For any other purpose, avoid these insecure functions and use Python’s other dynamic and metaprogramming features instead.