What’s New in This Chapter

This chapter is new in the second edition of Fluent Python. Let’s start with overloads.

Python functions may accept different combinations of arguments. The @typing.overload decorator allows annotating those different combinations. This is particularly important when the return type of the function depends on the type of two or more parameters.

>>> help(sum)
sum(iterable, /, start=0)
    Return the sum of a 'start' value (default: 0) plus an iterable of numbers

    When the iterable is empty, return the start value.
    This function is intended specifically for use with numeric values and may
    reject non-numeric types.

The sum built-in is written in C, but typeshed has overloaded type hints for it, in builtins.pyi:

@overload
def sum(__iterable: Iterable[_T]) -> Union[_T, int]: ...
@overload
def sum(__iterable: Iterable[_T], start: _S) -> Union[_T, _S]: ...

First let’s look at the overall syntax of overloads. That’s all the code about the sum you’ll find in the stub file (.pyi). The implementation would be in a different file. The ellipsis (...) has no function other than to fulfill the syntactic requirement for a function body, similar to pass. So .pyi files are valid Python files.

As mentioned in “Annotating Positional Only and Variadic Parameters”, the two leading underscores in __iterable are a PEP 484 convention for positional-only arguments that is enforced by Mypy. It means you can call sum(my_list), but not sum(__iterable = my_list).

The type checker tries to match the given arguments with each overloaded signature, in order. The call sum(range(100), 1000) doesn’t match the first overload, because that signature has only one parameter. But it matches the second.

You can also use @overload in a regular Python module, by writing the overloaded signatures right before the function’s actual signature and implementation. Example 15-1 shows how sum would appear annotated and implemented in a Python module.

import functools
import operator
from collections.abc import Iterable
from typing import overload, Union, TypeVar

T = TypeVar('T')
S = TypeVar('S')  

@overload
def sum(it: Iterable[T]) -> Union[T, int]: ...  
@overload
def sum(it: Iterable[T], /, start: S) -> Union[T, S]: ...  
def sum(it, /, start=0):  
    return functools.reduce(operator.add, it, start)

We need this second TypeVar in the second overload.

This signature is for the simple case: sum(my_iterable). The result type may be T—the type of the elements that my_iterable yields—or it may be int if the iterable is empty, because the default value of the start parameter is 0.

When start is given, it can be of any type S, so the result type is Union[T, S]. This is why we need S. If we reused T, then the type of start would have to be the same type as the elements of Iterable[T].

The signature of the actual function implementation has no type hints.

That’s a lot of lines to annotate a one-line function. Probably overkill, I know. At least it wasn’t a foo function.

If you want to learn about @overload by reading code, typeshed has hundreds of examples. On typeshed, the stub file for Python’s built-ins has 186 overloads as I write this—more than any other in the standard library.

The handy APIs we call Pythonic are often hard to annotate. In the next section we’ll see an example: six overloads are needed to properly annotate the flexible max built-in function.

TypeError: '>' not supported between instances of 'int' and 'NoneType'

# imports and definitions omitted, see next listing

MISSING = object()
EMPTY_MSG = 'max() arg is an empty sequence'

# overloaded type hints omitted, see next listing

def max(first, *args, key=None, default=MISSING):
    if args:
        series = args
        candidate = first
    else:
        series = iter(first)
        try:
            candidate = next(series)
        except StopIteration:
            if default is not MISSING:
                return default
            raise ValueError(EMPTY_MSG) from None
    if key is None:
        for current in series:
            if candidate < current:
                candidate = current
    else:
        candidate_key = key(candidate)
        for current in series:
            current_key = key(current)
            if candidate_key < current_key:
                candidate = current
                candidate_key = current_key
    return candidate

from collections.abc import Callable, Iterable
from typing import Protocol, Any, TypeVar, overload, Union

class SupportsLessThan(Protocol):
    def __lt__(self, other: Any) -> bool: ...

T = TypeVar('T')
LT = TypeVar('LT', bound=SupportsLessThan)
DT = TypeVar('DT')

MISSING = object()
EMPTY_MSG = 'max() arg is an empty sequence'

@overload
def max(__arg1: LT, __arg2: LT, *args: LT, key: None = ...) -> LT:
    ...
@overload
def max(__arg1: T, __arg2: T, *args: T, key: Callable[[T], LT]) -> T:
    ...
@overload
def max(__iterable: Iterable[LT], *, key: None = ...) -> LT:
    ...
@overload
def max(__iterable: Iterable[T], *, key: Callable[[T], LT]) -> T:
    ...
@overload
def max(__iterable: Iterable[LT], *, key: None = ...,
        default: DT) -> Union[LT, DT]:
    ...
@overload
def max(__iterable: Iterable[T], *, key: Callable[[T], LT],
        default: DT) -> Union[T, DT]:
    ...

@overload
def max(__arg1: LT, __arg2: LT, *_args: LT, key: None = ...) -> LT:
    ...
# ... lines omitted ...
@overload
def max(__iterable: Iterable[LT], *, key: None = ...) -> LT:
    ...

max(1, 2, -3)  # returns 2
max(['Go', 'Python', 'Rust'])  # returns 'Rust'

@overload
def max(__arg1: T, __arg2: T, *_args: T, key: Callable[[T], LT]) -> T:
    ...
# ... lines omitted ...
@overload
def max(__iterable: Iterable[T], *, key: Callable[[T], LT]) -> T:
    ...

max(1, 2, -3, key=abs)  # returns -3
max(['Go', 'Python', 'Rust'], key=len)  # returns 'Python'

@overload
def max(__iterable: Iterable[LT], *, key: None = ...,
        default: DT) -> Union[LT, DT]:
    ...

max([1, 2, -3], default=0)  # returns 2
max([], default=None)  # returns None

@overload
def max(__iterable: Iterable[T], *, key: Callable[[T], LT],
        default: DT) -> Union[T, DT]:
    ...

max([1, 2, -3], key=abs, default=None)  # returns -3
max([], key=abs, default=None)  # returns None

mymax_demo.py:109: error: Value of type variable "_LT" of "max"
  cannot be "None"

Python dictionaries are sometimes used as records, with the keys used as field names and field values of different types.

For example, consider a record describing a book in JSON or Python:

{"isbn": "0134757599",
 "title": "Refactoring, 2e",
 "authors": ["Martin Fowler", "Kent Beck"],
 "pagecount": 478}

Before Python 3.8, there was no good way to annotate a record like that, because the mapping types we saw in “Generic Mappings” limit all values to have the same type.

Here are two lame attempts to annotate a record like the preceding JSON object:

Dict[str, Any]

The values may be of any type.

Dict[str, Union[str, int, List[str]]]

Hard to read, and doesn’t preserve the relationship between field names and their respective field types: title is supposed to be a str, it can’t be an int or a List[str].

PEP 589—TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys addressed that problem. Example 15-4 shows a simple TypedDict.

from typing import TypedDict

class BookDict(TypedDict):
    isbn: str
    title: str
    authors: list[str]
    pagecount: int

At first glance, typing.TypedDict may seem like a data class builder, similar to typing.NamedTuple—covered in Chapter 5.

The syntactic similarity is misleading. TypedDict is very different. It exists only for the benefit of type checkers, and has no runtime effect.

TypedDict provides two things:

• Class-like syntax to annotate a dict with type hints for the value of each “field.”

• A constructor that tells the type checker to expect a dict with the keys and values as specified.

At runtime, a TypedDict constructor such as BookDict is a placebo: it has the same effect as calling the dict constructor with the same arguments.

The fact that BookDict creates a plain dict also means that:

• The “fields” in the pseudoclass definition don’t create instance attributes.

• You can’t write initializers with default values for the “fields.”

• Method definitions are not allowed.

Let’s explore the behavior of a BookDict at runtime (Example 15-5).

>>> from books import BookDict
>>> pp = BookDict(title='Programming Pearls',  
...               authors='Jon Bentley',  
...               isbn='0201657880',
...               pagecount=256)
>>> pp  
{'title': 'Programming Pearls', 'authors': 'Jon Bentley', 'isbn': '0201657880',
 'pagecount': 256}
>>> type(pp)
<class 'dict'>
>>> pp.title  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'title'
>>> pp['title']
'Programming Pearls'
>>> BookDict.__annotations__  
{'isbn': <class 'str'>, 'title': <class 'str'>, 'authors': typing.List[str],
 'pagecount': <class 'int'>}

You can call BookDict like a dict constructor with keyword arguments, or passing a dict argument—including a dict literal.

Oops…I forgot authors takes a list. But gradual typing means no type checking at runtime.

The result of calling BookDict is a plain dict…

…therefore you can’t read the data using object.field notation.

The type hints are in BookDict.__annotations__, and not in pp.

Without a type checker, TypedDict is as useful as comments: it may help people read the code, but that’s it. In contrast, the class builders from Chapter 5 are useful even if you don’t use a type checker, because at runtime they generate or enhance a custom class that you can instantiate. They also provide several useful methods or functions listed in Table 5-1.

Example 15-6 builds a valid BookDict and tries some operations on it. This shows how TypedDict enables Mypy to catch errors, shown in Example 15-7.

from books import BookDict
from typing import TYPE_CHECKING

def demo() -> None:  
    book = BookDict(  
        isbn='0134757599',
        title='Refactoring, 2e',
        authors=['Martin Fowler', 'Kent Beck'],
        pagecount=478
    )
    authors = book['authors'] 
    if TYPE_CHECKING:  
        reveal_type(authors)  
    authors = 'Bob'  
    book['weight'] = 4.2
    del book['title']


if __name__ == '__main__':
    demo()

Remember to add a return type, so that Mypy doesn’t ignore the function.

This is a valid BookDict: all the keys are present, with values of the correct types.

Mypy will infer the type of authors from the annotation for the 'authors' key in BookDict.

typing.TYPE_CHECKING is only True when the program is being type checked. At runtime, it’s always false.

The previous if statement prevents reveal_type(authors) from being called at runtime. reveal_type is not a runtime Python function, but a debugging facility provided by Mypy. That’s why there is no import for it. See its output in Example 15-7.

The last three lines of the demo function are illegal. They will cause error messages in Example 15-7.

Type checking demo_books.py from Example 15-6, we get Example 15-7.

…/typeddict/ $ mypy demo_books.py
demo_books.py:13: note: Revealed type is 'built-ins.list[built-ins.str]'  
demo_books.py:14: error: Incompatible types in assignment
                  (expression has type "str", variable has type "List[str]")  
demo_books.py:15: error: TypedDict "BookDict" has no key 'weight'  
demo_books.py:16: error: Key 'title' of TypedDict "BookDict" cannot be deleted  
Found 3 errors in 1 file (checked 1 source file)

This note is the result of reveal_type(authors).

The type of the authors variable was inferred from the type of the book['authors'] expression that initialized it. You can’t assign a str to a variable of type List[str]. Type checkers usually don’t allow the type of a variable to change.³

Cannot assign to a key that is not part of the BookDict definition.

Cannot delete a key that is part of the BookDict definition.

Now let’s see BookDict used in function signatures, to type check function calls.

Imagine you need to generate XML from book records, similar to this:

<BOOK>
  <ISBN>0134757599</ISBN>
  <TITLE>Refactoring, 2e</TITLE>
  <AUTHOR>Martin Fowler</AUTHOR>
  <AUTHOR>Kent Beck</AUTHOR>
  <PAGECOUNT>478</PAGECOUNT>
</BOOK>

If you were writing MicroPython code to be embedded in a tiny microcontroller, you might write a function like what’s shown in Example 15-8.⁴

AUTHOR_ELEMENT = '<AUTHOR>{}</AUTHOR>'

def to_xml(book: BookDict) -> str:  
    elements: list[str] = []  
    for key, value in book.items():
        if isinstance(value, list):  
            elements.extend(
                AUTHOR_ELEMENT.format(n) for n in value)  
        else:
            tag = key.upper()
            elements.append(f'<{tag}>{value}</{tag}>')
    xml = '\n\t'.join(elements)
    return f'<BOOK>\n\t{xml}\n</BOOK>'

The whole point of the example: using BookDict in the function signature.

It’s often necessary to annotate collections that start empty, otherwise Mypy can’t infer the type of the elements.⁵

Mypy understands isinstance checks, and treats value as a list in this block.

When I used key == 'authors' as the condition for the if guarding this block, Mypy found an error in this line: "object" has no attribute "__iter__", because it inferred the type of value returned from book.items() as object, which doesn’t support the __iter__ method required by the generator expression. With the isinstance check, this works because Mypy knows that value is a list in this block.

Example 15-9 shows a function that parses a JSON str and returns a BookDict.

def from_json(data: str) -> BookDict:
    whatever = json.loads(data)  
    return whatever  

The return type of json.loads() is Any.⁶

I can return whatever—of type Any—because Any is consistent-with every type, including the declared return type, BookDict.

The second point of Example 15-9 is very important to keep in mind: Mypy will not flag any problem in this code, but at runtime the value in whatever may not conform to the BookDict structure—in fact, it may not be a dict at all!

If you run Mypy with --disallow-any-expr, it will complain about the two lines in the body of from_json:

…/typeddict/ $ mypy books_any.py --disallow-any-expr
books_any.py:30: error: Expression has type "Any"
books_any.py:31: error: Expression has type "Any"
Found 2 errors in 1 file (checked 1 source file)

Lines 30 and 31 mentioned in the previous snippet are the body of the from_json function. We can silence the type error by adding a type hint to the initialization of the whatever variable, as in Example 15-10.

def from_json(data: str) -> BookDict:
    whatever: BookDict = json.loads(data)  
    return whatever  

--disallow-any-expr does not cause errors when an expression of type Any is immediately assigned to a variable with a type hint.

Now whatever is of type BookDict, the declared return type.

Static type checking is unable to prevent errors with code that is inherently dynamic, such as json.loads(), which builds Python objects of different types at runtime, as Examples 15-11, 15-12, and 15-13 demonstrate.

from books import to_xml, from_json
from typing import TYPE_CHECKING

def demo() -> None:
    NOT_BOOK_JSON = """
        {"title": "Andromeda Strain",
         "flavor": "pistachio",
         "authors": true}
    """
    not_book = from_json(NOT_BOOK_JSON)  
    if TYPE_CHECKING:  
        reveal_type(not_book)
        reveal_type(not_book['authors'])

    print(not_book)  
    print(not_book['flavor'])  

    xml = to_xml(not_book)  
    print(xml)  


if __name__ == '__main__':
    demo()

This line does not produce a valid BookDict—see the content of NOT_BOOK_JSON.

Let’s have Mypy reveal a couple of types.

This should not be a problem: print can handle object and every other type.

BookDict has no 'flavor' key, but the JSON source does…what will happen?

Remember the signature: def to_xml(book: BookDict) -> str:.

What will the XML output look like?

Now we check demo_not_book.py with Mypy (Example 15-12).

…/typeddict/ $ mypy demo_not_book.py
demo_not_book.py:12: note: Revealed type is
   'TypedDict('books.BookDict', {'isbn': built-ins.str,
                                 'title': built-ins.str,
                                 'authors': built-ins.list[built-ins.str],
                                 'pagecount': built-ins.int})'  
demo_not_book.py:13: note: Revealed type is 'built-ins.list[built-ins.str]'  
demo_not_book.py:16: error: TypedDict "BookDict" has no key 'flavor'  
Found 1 error in 1 file (checked 1 source file)

The revealed type is the nominal type, not the runtime content of not_book.

Again, this is the nominal type of not_book['authors'], as defined in BookDict. Not the runtime type.

This error is for line print(not_book['flavor']): that key does not exist in the nominal type.

Now let’s run demo_not_book.py, showing the output in Example 15-13.

…/typeddict/ $ python3 demo_not_book.py
{'title': 'Andromeda Strain', 'flavor': 'pistachio', 'authors': True}  
pistachio  
<BOOK>  
        <TITLE>Andromeda Strain</TITLE>
        <FLAVOR>pistachio</FLAVOR>
        <AUTHORS>True</AUTHORS>
</BOOK>

This is not really a BookDict.

The value of not_book['flavor'].

to_xml takes a BookDict argument, but there is no runtime checking: garbage in, garbage out.

Example 15-13 shows that demo_not_book.py outputs nonsense, but has no runtime errors. Using a TypedDict while handling JSON data did not provide much type safety.

If you look at the code for to_xml in Example 15-8 through the lens of duck typing, the argument book must provide an .items() method that returns an iterable of tuples like (key, value) where:

• key must have an .upper() method

• value can be anything

The point of this demonstration: when handling data with a dynamic structure, such as JSON or XML, TypedDict is absolutely not a replacement for data validation at runtime. For that, use pydantic.

TypedDict has more features, including support for optional keys, a limited form of inheritance, and an alternative declaration syntax. If you want to know more about it, please review PEP 589—TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys.

Now let’s turn our attention to a function that is best avoided, but sometimes is unavoidable: typing.cast.

No type system is perfect, and neither are the static type checkers, the type hints in the typeshed project, or the type hints in the third-party packages that have them.

The typing.cast() special function provides one way to handle type checking malfunctions or incorrect type hints in code we can’t fix. The Mypy 0.930 documentation explains:

Casts are used to silence spurious type checker warnings and give the type checker a little help when it can’t quite understand what is going on.

At runtime, typing.cast does absolutely nothing. This is its implementation:

def cast(typ, val):
    """Cast a value to a type.
    This returns the value unchanged.  To the type checker this
    signals that the return value has the designated type, but at
    runtime we intentionally don't check anything (we want this
    to be as fast as possible).
    """
    return val

PEP 484 requires type checkers to “blindly believe” the type stated in the cast. The “Casts” section of PEP 484 gives an example where the type checker needs the guidance of cast:

from typing import cast

def find_first_str(a: list[object]) -> str:
    index = next(i for i, x in enumerate(a) if isinstance(x, str))
    # We only get here if there's at least one string
    return cast(str, a[index])

The next() call on the generator expression will either return the index of a str item or raise StopIteration. Therefore, find_first_str will always return a str if no exception is raised, and str is the declared return type.

But if the last line were just return a[index], Mypy would infer the return type as object because the a argument is declared as list[object]. So the cast() is required to guide Mypy.⁷

Here is another example with cast, this time to correct an outdated type hint for Python’s standard library. In Example 21-12, I create an asyncio Server object and I want to get the address the server is listening to. I coded this line:

addr = server.sockets[0].getsockname()

But Mypy reported this error:

Value of type "Optional[List[socket]]" is not indexable

The type hint for Server.sockets on typeshed in May 2021 is valid for Python 3.6, where the sockets attribute could be None. But in Python 3.7, sockets became a property with a getter that always returns a list—which may be empty if the server has no sockets. And since Python 3.8, the getter returns a tuple (used as an immutable sequence).

Since I can’t fix typeshed right now,⁸ I added a cast, like this:

from asyncio.trsock import TransportSocket
from typing import cast

# ... many lines omitted ...

    socket_list = cast(tuple[TransportSocket, ...], server.sockets)
    addr = socket_list[0].getsockname()

Using cast in this case required a couple of hours to understand the problem and read asyncio source code to find the correct type of the sockets: the TransportSocket class from the undocumented asyncio.trsock module. I also had to add two import statements and another line of code for readability.⁹ But the code is safer.

The careful reader may note that sockets[0] could raise IndexError if sockets is empty. However, as far as I understand asyncio, that cannot happen in Example 21-12 because the server is ready to accept connections by the time I read its sockets attribute, therefore it will not be empty. Anyway, IndexError is a runtime error. Mypy can’t spot the problem even in a trivial case like print([][0]).

Despite the downsides, there are valid uses for cast. Here is something Guido van Rossum wrote about it:

What’s wrong with the occasional cast() call or # type: ignore comment?¹⁰

It is unwise to completely ban the use of cast, especially because the other workarounds are worse:

• # type: ignore is less informative.¹¹

• Using Any is contagious: since Any is consistent-with all types, abusing it may produce cascading effects through type inference, undermining the type checker’s ability to detect errors in other parts of the code.

Of course, not all typing mishaps can be fixed with cast. Sometimes we need # type: ignore, the occasional Any, or even leaving a function without type hints.

Next, let’s talk about using annotations at runtime.

At import time, Python reads the type hints in functions, classes, and modules, and stores them in attributes named __annotations__. For instance, consider the clip function in Example 15-14.¹²

def clip(text: str, max_len: int = 80) -> str:

The type hints are stored as a dict in the __annotations__ attribute of the function:

>>> from clip_annot import clip
>>> clip.__annotations__
{'text': <class 'str'>, 'max_len': <class 'int'>, 'return': <class 'str'>}

The 'return' key maps to the return type hint after the -> symbol in Example 15-14.

Note that the annotations are evaluated by the interpreter at import time, just as parameter default values are also evaluated. That’s why the values in the annotations are the Python classes str and int, and not the strings 'str' and 'int'. The import time evaluation of annotations is the standard as of Python 3.10, but that may change if PEP 563 or PEP 649 become the standard behavior.

class Rectangle:
    # ... lines omitted ...
    def stretch(self, factor: float) -> 'Rectangle':
        return Rectangle(width=self.width * factor)

from __future__ import annotations

>>> from clip_annot_post import clip
>>> clip.__annotations__
{'text': 'str', 'max_len': 'int', 'return': 'str'}

>>> from clip_annot_post import clip
>>> from typing import get_type_hints
>>> get_type_hints(clip)
{'text': <class 'str'>, 'max_len': <class 'int'>, 'return': <class 'str'>}

class Checked:
    @classmethod
    def _fields(cls) -> dict[str, type]:
        return get_type_hints(cls)
    # ... more lines ...

In Example 13-7 we defined the Tombola ABC: an interface for classes that work like a bingo cage. The LottoBlower class from Example 13-10 is a concrete implementation. Now we’ll study a generic version of LottoBlower used like in Example 15-15.

from generic_lotto import LottoBlower

machine = LottoBlower[int](range(1, 11))  

first = machine.pick()  
remain = machine.inspect()  

To instantiate a generic class, we give it an actual type parameter, like int here.

Mypy will correctly infer that first is an int…

… and that remain is a tuple of integers.

In addition, Mypy reports violations of the parameterized type with helpful messages, such as what’s shown in Example 15-16.

from generic_lotto import LottoBlower

machine = LottoBlower[int]([1, .2])
## error: List item 1 has incompatible type "float";  
##        expected "int"

machine = LottoBlower[int](range(1, 11))

machine.load('ABC')
## error: Argument 1 to "load" of "LottoBlower"  
##        has incompatible type "str";
##        expected "Iterable[int]"
## note:  Following member(s) of "str" have conflicts:
## note:      Expected:
## note:          def __iter__(self) -> Iterator[int]
## note:      Got:
## note:          def __iter__(self) -> Iterator[str]

Upon instantiation of LottoBlower[int], Mypy flags the float.

When calling .load('ABC'), Mypy explains why a str won’t do: str.__iter__ returns an Iterator[str], but LottoBlower[int] requires an Iterator[int].

Example 15-17 is the implementation.

import random

from collections.abc import Iterable
from typing import TypeVar, Generic

from tombola import Tombola

T = TypeVar('T')

class LottoBlower(Tombola, Generic[T]):  

    def __init__(self, items: Iterable[T]) -> None:  
        self._balls = list[T](items)

    def load(self, items: Iterable[T]) -> None:  
        self._balls.extend(items)

    def pick(self) -> T:  
        try:
            position = random.randrange(len(self._balls))
        except ValueError:
            raise LookupError('pick from empty LottoBlower')
        return self._balls.pop(position)

    def loaded(self) -> bool:  
        return bool(self._balls)

    def inspect(self) -> tuple[T, ...]:  
        return tuple(self._balls)

Generic class declarations often use multiple inheritance, because we need to subclass Generic to declare the formal type parameters—in this case, T.

The items argument in __init__ is of type Iterable[T], which becomes Iterable[int] when an instance is declared as LottoBlower[int].

The load method is likewise constrained.

The return type of T now becomes int in a LottoBlower[int].

No type variable here.

Finally, T sets the type of the items in the returned tuple.

Now that we’ve seen how to implement a generic class, let’s define the terminology to talk about generics.

We first saw the concept of variance in “Variance in Callable types”, applied to parameterized generic Callable types. Here we’ll expand the concept to cover generic collection types, using a “real world” analogy to make this abstract concept more concrete.

Imagine that a school cafeteria has a rule that only juice dispensers can be installed.¹⁵ General beverage dispensers are not allowed because they may serve sodas, which are banned by the school board.¹⁶

from typing import TypeVar, Generic

class Beverage:  
    """Any beverage."""

class Juice(Beverage):
    """Any fruit juice."""

class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""

T = TypeVar('T')  

class BeverageDispenser(Generic[T]):  
    """A dispenser parameterized on the beverage type."""
    def __init__(self, beverage: T) -> None:
        self.beverage = beverage

    def dispense(self) -> T:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:  
    """Install a fruit juice dispenser."""

juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)

beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
##          expected "BeverageDispenser[Juice]"

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[OrangeJuice]"
##          expected "BeverageDispenser[Juice]"

T_co = TypeVar('T_co', covariant=True)  


class BeverageDispenser(Generic[T_co]):  
    def __init__(self, beverage: T_co) -> None:
        self.beverage = beverage

    def dispense(self) -> T_co:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:  
    """Install a fruit juice dispenser."""

juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)

beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
##          expected "BeverageDispenser[Juice]"

from typing import TypeVar, Generic

class Refuse:  
    """Any refuse."""

class Biodegradable(Refuse):
    """Biodegradable refuse."""

class Compostable(Biodegradable):
    """Compostable refuse."""

T_contra = TypeVar('T_contra', contravariant=True)  

class TrashCan(Generic[T_contra]):  
    def put(self, refuse: T_contra) -> None:
        """Store trash until dumped."""

def deploy(trash_can: TrashCan[Biodegradable]):
    """Deploy a trash can for biodegradable refuse."""

bio_can: TrashCan[Biodegradable] = TrashCan()
deploy(bio_can)

trash_can: TrashCan[Refuse] = TrashCan()
deploy(trash_can)

compost_can: TrashCan[Compostable] = TrashCan()
deploy(compost_can)
## mypy: Argument 1 to "deploy" has
## incompatible type "TrashCan[Compostable]"
##          expected "TrashCan[Biodegradable]"

class list(MutableSequence[_T], Generic[_T]):
    @overload
    def __init__(self) -> None: ...
    @overload
    def __init__(self, iterable: Iterable[_T]) -> None: ...
    # ... lines omitted ...
    def append(self, __object: _T) -> None: ...
    def extend(self, __iterable: Iterable[_T]) -> None: ...
    def pop(self, __index: int = ...) -> _T: ...
    # etc...

class FrozenSet(frozenset, AbstractSet[T_co]):

           float :> int
frozenset[float] :> frozenset[int]

          Refuse :> Biodegradable
TrashCan[Refuse] <: TrashCan[Biodegradable]

The Python 3.10 standard library provides a few generic static protocols. One of them is SupportsAbs, implemented like this in the typing module:

@runtime_checkable
class SupportsAbs(Protocol[T_co]):
    """An ABC with one abstract method __abs__ that is covariant in its
        return type."""
    __slots__ = ()

    @abstractmethod
    def __abs__(self) -> T_co:
        pass

T_co is declared according to the naming convention:

T_co = TypeVar('T_co', covariant=True)

Thanks to SupportsAbs, Mypy recognizes this code as valid, as you can see in Example 15-21.

import math
from typing import NamedTuple, SupportsAbs

class Vector2d(NamedTuple):
    x: float
    y: float

    def __abs__(self) -> float:  
        return math.hypot(self.x, self.y)

def is_unit(v: SupportsAbs[float]) -> bool:  
    """'True' if the magnitude of 'v' is close to 1."""
    return math.isclose(abs(v), 1.0)  

assert issubclass(Vector2d, SupportsAbs)  

v0 = Vector2d(0, 1)  
sqrt2 = math.sqrt(2)
v1 = Vector2d(sqrt2 / 2, sqrt2 / 2)
v2 = Vector2d(1, 1)
v3 = complex(.5, math.sqrt(3) / 2)
v4 = 1  

assert is_unit(v0)
assert is_unit(v1)
assert not is_unit(v2)
assert is_unit(v3)
assert is_unit(v4)

print('OK')

Defining __abs__ makes Vector2d consistent-with SupportsAbs.

Parameterizing SupportsAbs with float ensures…

…that Mypy accepts abs(v) as the first argument for math.isclose.

Thanks to @runtime_checkable in the definition of SupportsAbs, this is a valid runtime assertion.

The remaining code all passes Mypy checks and runtime assertions.

The int type is also consistent-with SupportsAbs. According to typeshed, int.__abs__ returns an int, which is consistent-with the float type parameter declared in the is_unit type hint for the v argument.

Similarly, we can write a generic version of the RandomPicker protocol presented in Example 13-18, which was defined with a single method pick returning Any.

Example 15-22 shows how to make a generic RandomPicker covariant on the return type of pick.

from typing import Protocol, runtime_checkable, TypeVar

T_co = TypeVar('T_co', covariant=True)  

@runtime_checkable
class RandomPicker(Protocol[T_co]):  
    def pick(self) -> T_co: ...  

Declare T_co as covariant.

This makes RandomPicker generic with a covariant formal type parameter.

Use T_co as the return type.

The generic RandomPicker protocol can be covariant because its only formal parameter is used in a return type.

With this, we can call it a chapter.

The chapter started with a simple example of using @overload, followed by a much more complex example that we studied in detail: the overloaded signatures required to correctly annotate the max built-in function.

The typing.TypedDict special construct came next. I chose to cover it here, and not in Chapter 5 where we saw typing.NamedTuple, because TypedDict is not a class builder; it’s merely a way to add type hints to a variable or argument that requires a dict with a specific set of string keys, and specific types for each key—which happens when we use a dict as a record, often in the context of handling with JSON data. That section was a bit long because using TypedDict can give a false sense of security, and I wanted to show how runtime checks and error handling are really inevitable when trying to make statically structured records out of mappings that are dynamic in nature.

Next we talked about typing.cast, a function designed to let us guide the work of the type checker. It’s important to carefully consider when to use cast, because overusing it hinders the type checker.

Runtime access to type hints came next. The key point was to use typing.​get_type_hints instead of reading the __annotations__ attribute directly. However, that function may be unreliable with some annotations, and we saw that Python core developers are still working on a way to make type hints usable at runtime, while reducing their impact on CPU and memory usage.

The final sections were about generics, starting with the LottoBlower generic class—which we later learned is an invariant generic class. That example was followed by definitions of four basic terms: generic type, formal type parameter, parameterized type, and actual type parameter.

The major topic of variance was presented next, using cafeteria beverage dispensers and trash cans as “real life” examples of invariant, covariant, and contravariant generic types. Next we reviewed, formalized, and further applied those concepts to examples in Python’s standard library.

Lastly, we saw how a generic static protocol is defined, first considering the typing.SupportsAbs protocol, and then applying the same idea to the RandomPicker example, making it more strict than the original protocol from Chapter 13.

Further Reading

Python’s static type system was complex as initially designed, and is getting more complex with each passing year. Table 15-1 lists all the PEPs that I am aware of as of May 2021. It would take a whole book to cover everything.

Python’s official documentation hardly keeps up with all that, so Mypy’s documentation is an essential reference. Robust Python by Patrick Viafore (O’Reilly) is the first book with extensive coverage of Python’s static type system that I know about, published August 2021. You may be reading the second such book right now.

The subtle topic of variance has its own section in PEP 484, and is also covered in the “Generics” page of Mypy, as well as in its invaluable “Common Issues” page.

PEP 362—Function Signature Object is worth reading if you intend to use the inspect module that complements the typing.get_type_hints function.

If you are interested in the history of Python, you may like to know that Guido van Rossum posted “Adding Optional Static Typing to Python” on December 23, 2004.

“Python 3 Types in the Wild: A Tale of Two Type Systems” is a research paper by Ingkarat Rak-amnouykit and others from the Rensselaer Polytechnic Institute and IBM TJ Watson Research Center. The paper surveys the use of type hints in open source projects on GitHub, showing that most projects don’t use them, and also that most projects that have type hints apparently don’t use a type checker. I found most interesting the discussion of the different semantics of Mypy and Google’s pytype, which they conclude are “essentially two different type systems.”

Two seminal papers about gradual typing are Gilad Bracha’s “Pluggable Type Systems”, and “Static Typing Where Possible, Dynamic Typing When Needed: The End of the Cold War Between Programming Languages” by Eric Meijer and Peter Drayton.¹⁷

I learned a lot reading the relevant parts of some books about other languages that implement some of the same ideas:

• Atomic Kotlin by Bruce Eckel and Svetlana Isakova (Mindview)

• Effective Java, 3rd ed., by Joshua Bloch (Addison-Wesley)

• Programming with Types: TypeScript Examples by Vlad Riscutia (Manning)

• Programming TypeScript by Boris Cherny (O’Reilly)

• The Dart Programming Language by Gilad Bracha (Addison-Wesley)¹⁸

For some critical views on type systems, I recommend Victor Youdaiken’s posts “Bad ideas in type theory” and “Types considered harmful II”.

Finally, I was surprised to find “Generics Considered Harmful” by Ken Arnold, a core contributor to Java from the beginning, as well as coauthor of the first four editions of the official The Java Programming Language book (Addison-Wesley)—in collaboration with James Gosling, the lead designer of Java.

Sadly, Arnold’s criticism applies to Python’s static type system as well. While reading the many rules and special cases of the typing PEPs, I was constantly reminded of this passage from Gosling’s post:

Which brings up the problem that I always cite for C++: I call it the “N^th order exception to the exception rule.” It sounds like this: “You can do x, except in case y, unless y does z, in which case you can if …”

Fortunately, Python has a key advantage over Java and C++: an optional type system. We can squelch type checkers and omit type hints when they become too cumbersome.