What’s New in This Chapter

All the code in the “Class Metaprogramming” chapter of the first edition of Fluent Python still runs correctly. However, some of the previous examples no longer represent the simplest solutions in light of new features added since Python 3.6.

“Metaclasses in the Real World” is a new section with some high-level considerations about the applicability of metaclasses.

We’ll get started by reviewing attributes and methods defined in the Python Data Model for all classes.

Classes as Objects

Like most program entities in Python, classes are also objects. Every class has a number of attributes defined in the Python Data Model, documented in “4.13. Special Attributes” of the “Built-in Types” chapter in The Python Standard Library. Three of those attributes appeared several times in this book already: __class__, __name__, and __mro__. Other class standard attributes are:

cls.__bases__

The tuple of base classes of the class.

cls.__qualname__

The qualified name of a class or function, which is a dotted path from the global scope of the module to the class definition. This is relevant when the class is defined inside another class. For example, in a Django model class such as Ox, there is an inner class called Meta. The __qualname__ of Meta is Ox.Meta, but its __name__ is just Meta. The specification for this attribute is PEP 3155—Qualified name for classes and functions.

cls.__subclasses__()

This method returns a list of the immediate subclasses of the class. The implementation uses weak references to avoid circular references between the superclass and its subclasses—which hold a strong reference to the superclasses in their __bases__ attribute. The method lists subclasses currently in memory. Subclasses in modules not yet imported will not appear in the result.

cls.mro()

The interpreter calls this method when building a class to obtain the tuple of superclasses stored in the __mro__ attribute of the class. A metaclass can override this method to customize the method resolution order of the class under construction.

We usually think of type as a function that returns the class of an object, because that’s what type(my_object) does: it returns my_object.__class__.

class MyClass(MySuperClass, MyMixin):
    x = 42

    def x2(self):
        return self.x * 2

MyClass = type('MyClass',
               (MySuperClass, MyMixin),
               {'x': 42, 'x2': lambda self: self.x * 2},
          )

name

The identifier that appears after the class keyword, e.g., MyClass.

bases

The tuple of superclasses given in parentheses after the class identifier, or (object,) if superclasses are not mentioned in the class statement.

dict

A mapping of attribute names to values. Callables become methods, as we saw in “Methods Are Descriptors”. Other values become class attributes.

The type class is a metaclass: a class that builds classes. In other words, instances of the type class are classes. The standard library provides a few other metaclasses, but type is the default:

>>> type(7)
<class 'int'>
>>> type(int)
<class 'type'>
>>> type(OSError)
<class 'type'>
>>> class Whatever:
...     pass
...
>>> type(Whatever)
<class 'type'>

We’ll build custom metaclasses in “Metaclasses 101”.

Next, we’ll use the type built-in to make a function that builds classes.

A Class Factory Function

The standard library has a class factory function that appears several times in this book: collections.namedtuple. In Chapter 5 we also saw typing.NamedTuple and @dataclass. All of these class builders leverage techniques covered in this chapter.

We’ll start with a super simple factory for classes of mutable objects—the simplest possible replacement for @dataclass.

Suppose I’m writing a pet shop application and I want to store data for dogs as simple records. But I don’t want to write boilerplate like this:

class Dog:
    def __init__(self, name, weight, owner):
        self.name = name
        self.weight = weight
        self.owner = owner

Boring…each field name appears three times, and that boilerplate doesn’t even buy us a nice repr:

>>> rex = Dog('Rex', 30, 'Bob')
>>> rex
<__main__.Dog object at 0x2865bac>

Taking a hint from collections.namedtuple, let’s create a record_factory that creates simple classes like Dog on the fly. Example 24-1 shows how it should work.

    >>> Dog = record_factory('Dog', 'name weight owner')  
    >>> rex = Dog('Rex', 30, 'Bob')
    >>> rex  
    Dog(name='Rex', weight=30, owner='Bob')
    >>> name, weight, _ = rex  
    >>> name, weight
    ('Rex', 30)
    >>> "{2}'s dog weighs {1}kg".format(*rex)  
    "Bob's dog weighs 30kg"
    >>> rex.weight = 32  
    >>> rex
    Dog(name='Rex', weight=32, owner='Bob')
    >>> Dog.__mro__  
    (<class 'factories.Dog'>, <class 'object'>)

Factory can be called like namedtuple: class name, followed by attribute names separated by spaces in a single string.

Nice repr.

Instances are iterable, so they can be conveniently unpacked on assignment…

…or when passing to functions like format.

A record instance is mutable.

The newly created class inherits from object—no relationship to our factory.

The code for record_factory is in Example 24-2.³

from typing import Union, Any
from collections.abc import Iterable, Iterator

FieldNames = Union[str, Iterable[str]]  

def record_factory(cls_name: str, field_names: FieldNames) -> type[tuple]:  

    slots = parse_identifiers(field_names)  

    def __init__(self, *args, **kwargs) -> None:  
        attrs = dict(zip(self.__slots__, args))
        attrs.update(kwargs)
        for name, value in attrs.items():
            setattr(self, name, value)

    def __iter__(self) -> Iterator[Any]:  
        for name in self.__slots__:
            yield getattr(self, name)

    def __repr__(self):  
        values = ', '.join(f'{name}={value!r}'
            for name, value in zip(self.__slots__, self))
        cls_name = self.__class__.__name__
        return f'{cls_name}({values})'

    cls_attrs = dict(  
        __slots__=slots,
        __init__=__init__,
        __iter__=__iter__,
        __repr__=__repr__,
    )

    return type(cls_name, (object,), cls_attrs)  


def parse_identifiers(names: FieldNames) -> tuple[str, ...]:
    if isinstance(names, str):
        names = names.replace(',', ' ').split()  
    if not all(s.isidentifier() for s in names):
        raise ValueError('names must all be valid identifiers')
    return tuple(names)

User can provide field names as a single string or an iterable of strings.

Accept arguments like the first two of collections.namedtuple; return a type—i.e., a class that behaves like a tuple.

Build a tuple of attribute names; this will be the __slots__ attribute of the new class.

This function will become the __init__ method in the new class. It accepts positional and/or keyword arguments.⁴

Yield the field values in the order given by __slots__.

Produce the nice repr, iterating over __slots__ and self.

Assemble a dictionary of class attributes.

Build and return the new class, calling the type constructor.

Convert names separated by spaces or commas to list of str.

Example 24-2 is the first time we’ve seen type in a type hint. If the annotation was just -> type, that would mean that record_factory returns a class—and it would be correct. But the annotation -> type[tuple] is more precise: it says the returned class will be a subclass of tuple.

The last line of record_factory in Example 24-2 builds a class named by the value of cls_name, with object as its single immediate base class, and with a namespace loaded with __slots__, __init__, __iter__, and __repr__, of which the last three are instance methods.

We could have named the __slots__ class attribute anything else, but then we’d have to implement __setattr__ to validate the names of attributes being assigned, because for our record-like classes we want the set of attributes to be always the same and in the same order. However, recall that the main feature of __slots__ is saving memory when you are dealing with millions of instances, and using __slots__ has some drawbacks, discussed in “Saving Memory with __slots__”.

Now let’s see how to emulate more modern class builders like typing.NamedTuple, which takes a user-defined class written as a class statement, and automatically enhances it with more functionality.

Both __init_subclass__ and __set_name__ were proposed in PEP 487—Simpler customization of class creation. We saw the __set_name__ special method for descriptors for the first time in “LineItem Take #4: Automatic Naming of Storage Attributes”. Now let’s study __init_subclass__.

In Chapter 5, we saw that typing.NamedTuple and @dataclass let programmers use the class statement to specify attributes for a new class, which is then enhanced by the class builder with the automatic addition of essential methods like __init__, __repr__, __eq__, etc.

Both of these class builders read type hints in the user’s class statement to enhance the class. Those type hints also allow static type checkers to validate code that sets or gets those attributes. However, NamedTuple and @dataclass do not take advantage of the type hints for attribute validation at runtime. The Checked class in the next example does.

Example 24-3 shows how to use Checked to build a Movie class.

    >>> class Movie(Checked):  
    ...     title: str  
    ...     year: int
    ...     box_office: float
    ...
    >>> movie = Movie(title='The Godfather', year=1972, box_office=137)  
    >>> movie.title
    'The Godfather'
    >>> movie  
    Movie(title='The Godfather', year=1972, box_office=137.0)

Movie inherits from Checked—which we’ll define later in Example 24-5.

Each attribute is annotated with a constructor. Here I used built-in types.

Movie instances must be created using keyword arguments.

In return, you get a nice __repr__.

The constructors used as the attribute type hints may be any callable that takes zero or one argument and returns a value suitable for the intended field type, or rejects the argument by raising TypeError or ValueError.

Using built-in types for the annotations in Example 24-3 means the values must be acceptable by the constructor of the type. For int, this means any x such that int(x) returns an int. For str, anything goes at runtime, because str(x) works with any x in Python.⁵

When called with no arguments, the constructor should return a default value of its type.⁶

This is standard behavior for Python’s built-in constructors:

>>> int(), float(), bool(), str(), list(), dict(), set()
(0, 0.0, False, '', [], {}, set())

In a Checked subclass like Movie, missing parameters create instances with default values returned by the field constructors. For example:

    >>> Movie(title='Life of Brian')
    Movie(title='Life of Brian', year=0, box_office=0.0)

The constructors are used for validation during instantiation and when an attribute is set directly on an instance:

    >>> blockbuster = Movie(title='Avatar', year=2009, box_office='billions')
    Traceback (most recent call last):
      ...
    TypeError: 'billions' is not compatible with box_office:float
    >>> movie.year = 'MCMLXXII'
    Traceback (most recent call last):
      ...
    TypeError: 'MCMLXXII' is not compatible with year:int

movie.year = 'MCMLXXII'

blockbuster = Movie(title='Avatar', year='MMIX')

Now let’s look at the implementation of checkedlib.py. The first class is the Field descriptor, as shown in Example 24-4.

from collections.abc import Callable  
from typing import Any, NoReturn, get_type_hints


class Field:
    def __init__(self, name: str, constructor: Callable) -> None:  
        if not callable(constructor) or constructor is type(None):  
            raise TypeError(f'{name!r} type hint must be callable')
        self.name = name
        self.constructor = constructor

    def __set__(self, instance: Any, value: Any) -> None:
        if value is ...:  
            value = self.constructor()
        else:
            try:
                value = self.constructor(value)  
            except (TypeError, ValueError) as e:  
                type_name = self.constructor.__name__
                msg = f'{value!r} is not compatible with {self.name}:{type_name}'
                raise TypeError(msg) from e
        instance.__dict__[self.name] = value  

Recall that since Python 3.9, the Callable type for annotations is the ABC in collections.abc, and not the deprecated typing.Callable.

This is a minimal Callable type hint; the parameter type and return type for constructor are both implicitly Any.

For runtime checking, we use the callable built-in.⁷ The test against type(None) is necessary because Python reads None in a type as NoneType, the class of None (therefore callable), but a useless constructor that only returns None.

If Checked.__init__ sets the value as ... (the Ellipsis built-in object), we call the constructor with no arguments.

Otherwise, call the constructor with the given value.

If constructor raises either of these exceptions, we raise TypeError with a helpful message including the names of the field and constructor; e.g., 'MMIX' is not compatible with year:int.

If no exceptions were raised, the value is stored in the instance.__dict__.

In __set__, we need to catch TypeError and ValueError because built-in constructors may raise either of them, depending on the argument. For example, float(None) raises TypeError, but float('A') raises ValueError. On the other hand, float('8') raises no error and returns 8.0. I hereby declare that this is a feature and not a bug of this toy example.

Now let’s focus on the Checked class. I split it in two listings. Example 24-5 shows the top of the class, which includes the most important methods in this example. The remaining methods are in Example 24-6.

class Checked:
    @classmethod
    def _fields(cls) -> dict[str, type]:  
        return get_type_hints(cls)

    def __init_subclass__(subclass) -> None:  
        super().__init_subclass__()           
        for name, constructor in subclass._fields().items():   
            setattr(subclass, name, Field(name, constructor))  

    def __init__(self, **kwargs: Any) -> None:
        for name in self._fields():             
            value = kwargs.pop(name, ...)       
            setattr(self, name, value)          
        if kwargs:                              
            self.__flag_unknown_attrs(*kwargs)  

I wrote this class method to hide the call to typing.get_type_hints from the rest of the class. If I need to support Python ≥ 3.10 only, I’d call inspect.get_annotations instead. Review “Problems with Annotations at Runtime” for the issues with those functions.

__init_subclass__ is called when a subclass of the current class is defined. It gets that new subclass as its first argument—which is why I named the argument subclass instead of the usual cls. For more on this, see “__init_subclass__ Is Not a Typical Class Method”.

super().__init_subclass__() is not strictly necessary, but should be invoked to play nice with other classes that might implement .__init_subclass__() in the same inheritance graph. See “Multiple Inheritance and Method Resolution Order”.

Iterate over each field name and constructor…

…creating an attribute on subclass with that name bound to a Field descriptor parameterized with name and constructor.

For each name in the class fields…

…get the corresponding value from kwargs and remove it from kwargs. Using ... (the Ellipsis object) as default allows us to distinguish between arguments given the value None from arguments that were not given.⁸

This setattr call triggers Checked.__setattr__, shown in Example 24-6.

If there are remaining items in kwargs, their names do not match any of the declared fields, and __init__ will fail.

The error is reported by __flag_unknown_attrs, listed in Example 24-6. It takes a *names argument with the unknown attribute names. I used a single asterisk in *kwargs to pass its keys as a sequence of arguments.

Now let’s see the remaining methods of the Checked class, continuing from Example 24-5. Note that I prepended _ to the _fields and _asdict method names for the same reason the collections.namedtuple API does: to reduce the chance of name clashes with user-defined field names.

    def __setattr__(self, name: str, value: Any) -> None:  
        if name in self._fields():              
            cls = self.__class__
            descriptor = getattr(cls, name)
            descriptor.__set__(self, value)     
        else:                                   
            self.__flag_unknown_attrs(name)

    def __flag_unknown_attrs(self, *names: str) -> NoReturn:  
        plural = 's' if len(names) > 1 else ''
        extra = ', '.join(f'{name!r}' for name in names)
        cls_name = repr(self.__class__.__name__)
        raise AttributeError(f'{cls_name} object has no attribute{plural} {extra}')

    def _asdict(self) -> dict[str, Any]:  
        return {
            name: getattr(self, name)
            for name, attr in self.__class__.__dict__.items()
            if isinstance(attr, Field)
        }

    def __repr__(self) -> str:  
        kwargs = ', '.join(
            f'{key}={value!r}' for key, value in self._asdict().items()
        )
        return f'{self.__class__.__name__}({kwargs})'

Intercept all attempts to set an instance attribute. This is needed to prevent setting an unknown attribute.

If the attribute name is known, fetch the corresponding descriptor.

Usually we don’t need to call the descriptor __set__ explicitly. It was necessary in this case because __setattr__ intercepts all attempts to set an attribute on the instance, including in the presence of an overriding descriptor such as Field.⁹

Otherwise, the attribute name is unknown, and an exception will be raised by __flag_unknown_attrs.

Build a helpful error message listing all unexpected arguments, and raise AttributeError. This is a rare example of the NoReturn special type, covered in “NoReturn”.

Create a dict from the attributes of a Movie object. I’d call this method _as_dict, but I followed the convention started by the _asdict method in collections.namedtuple.

Implementing a nice __repr__ is the main reason for having _asdict in this example.

The Checked example illustrates how to handle overriding descriptors when implementing __setattr__ to block arbitrary attribute setting after instantiation. It is debatable whether implementing __setattr__ is worthwhile in this example. Without it, setting movie.director = 'Greta Gerwig' would succeed, but the director attribute would not be checked in any way, and would not appear in the __repr__ nor would it be included in the dict returned by _asdict—both defined in Example 24-6.

In record_factory.py (Example 24-2) I solved this issue using the __slots__ class attribute. However, this simpler solution is not viable in this case, as explained next.

A class decorator is a callable that behaves similarly to a function decorator: it gets the decorated class as an argument, and should return a class to replace the decorated class. Class decorators often return the decorated class itself, after injecting more methods in it via attribute assignment.

Probably the most common reason to choose a class decorator over the simpler __init_subclass__ is to avoid interfering with other class features, such as inheritance and metaclasses.10

In this section, we’ll study checkeddeco.py, which provides the same service as checkedlib.py, but using a class decorator. As usual, we’ll start by looking at a usage example, extracted from the doctests in checkeddeco.py (Example 24-7).

    >>> @checked
    ... class Movie:
    ...     title: str
    ...     year: int
    ...     box_office: float
    ...
    >>> movie = Movie(title='The Godfather', year=1972, box_office=137)
    >>> movie.title
    'The Godfather'
    >>> movie
    Movie(title='The Godfather', year=1972, box_office=137.0)

The only difference between Example 24-7 and Example 24-3 is the way the Movie class is declared: it is decorated with @checked instead of subclassing Checked. Otherwise, the external behavior is the same, including the type validation and default value assignments shown after Example 24-3 in “Introducing __init_subclass__”.

Now let’s look at the implementation of checkeddeco.py. The imports and Field class are the same as in checkedlib.py, listed in Example 24-4. There is no other class, only functions in checkeddeco.py.

The logic previously implemented in __init_subclass__ is now part of the checked function—the class decorator listed in Example 24-8.

def checked(cls: type) -> type:  
    for name, constructor in _fields(cls).items():    
        setattr(cls, name, Field(name, constructor))  

    cls._fields = classmethod(_fields)  # type: ignore  

    instance_methods = (  
        __init__,
        __repr__,
        __setattr__,
        _asdict,
        __flag_unknown_attrs,
    )
    for method in instance_methods:  
        setattr(cls, method.__name__, method)

    return cls  

Recall that classes are instances of type. These type hints strongly suggest this is a class decorator: it takes a class and returns a class.

_fields is a top-level function defined later in the module (in Example 24-9).

Replacing each attribute returned by _fields with a Field descriptor instance is what __init_subclass__ did in Example 24-5. Here there is more work to do…

Build a class method from _fields, and add it to the decorated class. The type: ignore comment is needed because Mypy complains that type has no _fields attribute.

Module-level functions that will become instance methods of the decorated class.

Add each of the instance_methods to cls.

Return the decorated cls, fulfilling the essential contract of a class decorator.

Every top-level function in checkeddeco.py is prefixed with an underscore, except the checked decorator. This naming convention makes sense for a couple of reasons:

• checked is part of the public interface of the checkeddeco.py module, but the other functions are not.

• The functions in Example 24-9 will be injected in the decorated class, and the leading _ reduces the chance of naming conflicts with user-defined attributes and methods of the decorated class.

The rest of checkeddeco.py is listed in Example 24-9. Those module-level functions have the same code as the corresponding methods of the Checked class of checkedlib.py. They were explained in Examples 24-5 and 24-6.

Note that the _fields function does double duty in checkeddeco.py. It is used as a regular function in the first line of the checked decorator, and it will also be injected as a class method of the decorated class.

def _fields(cls: type) -> dict[str, type]:
    return get_type_hints(cls)

def __init__(self: Any, **kwargs: Any) -> None:
    for name in self._fields():
        value = kwargs.pop(name, ...)
        setattr(self, name, value)
    if kwargs:
        self.__flag_unknown_attrs(*kwargs)

def __setattr__(self: Any, name: str, value: Any) -> None:
    if name in self._fields():
        cls = self.__class__
        descriptor = getattr(cls, name)
        descriptor.__set__(self, value)
    else:
        self.__flag_unknown_attrs(name)

def __flag_unknown_attrs(self: Any, *names: str) -> NoReturn:
    plural = 's' if len(names) > 1 else ''
    extra = ', '.join(f'{name!r}' for name in names)
    cls_name = repr(self.__class__.__name__)
    raise AttributeError(f'{cls_name} has no attribute{plural} {extra}')

def _asdict(self: Any) -> dict[str, Any]:
    return {
        name: getattr(self, name)
        for name, attr in self.__class__.__dict__.items()
        if isinstance(attr, Field)
    }

def __repr__(self: Any) -> str:
    kwargs = ', '.join(
        f'{key}={value!r}' for key, value in self._asdict().items()
    )
    return f'{self.__class__.__name__}({kwargs})'

The checkeddeco.py module implements a simple but usable class decorator. Python’s @dataclass does a lot more. It supports many configuration options, adds more methods to the decorated class, handles or warns about conflicts with user-defined methods in the decorated class, and even traverses the __mro__ to collect user-defined attributes declared in the superclasses of the decorated class. The source code of the dataclasses package in Python 3.9 is more than 1,200 lines long.

For metaprogramming classes, we must be aware of when the Python interpreter evaluates each block of code during the construction of a class. This is covered next.

Python programmers talk about “import time” versus “runtime,” but the terms are not strictly defined and there is a gray area between them.

In particular, the import statement is not merely a declaration,11 but it actually runs all the top-level code of a module when it is imported for the first time in the process. Further imports of the same module will use a cache, and then the only effect will be binding the imported objects to names in the client module. That top-level code may do anything, including actions typical of “runtime,” such as writing to a log or connecting to a database.¹² That’s why the border between “import time” and “runtime” is fuzzy: the import statement can trigger all sorts of “runtime” behavior. Conversely, “import time” can also happen deep inside runtime, because the import statement and the __import__() built-in can be used inside any regular function.

This is all rather abstract and subtle, so let’s do some experiments to see what happens when.

print('@ builderlib module start')

class Builder:  
    print('@ Builder body')

    def __init_subclass__(cls):  
        print(f'@ Builder.__init_subclass__({cls!r})')

        def inner_0(self):  
            print(f'@ SuperA.__init_subclass__:inner_0({self!r})')

        cls.method_a = inner_0

    def __init__(self):
        super().__init__()
        print(f'@ Builder.__init__({self!r})')


def deco(cls):  
    print(f'@ deco({cls!r})')

    def inner_1(self):  
        print(f'@ deco:inner_1({self!r})')

    cls.method_b = inner_1
    return cls  

class Descriptor:  
    print('@ Descriptor body')

    def __init__(self):  
        print(f'@ Descriptor.__init__({self!r})')

    def __set_name__(self, owner, name):  
        args = (self, owner, name)
        print(f'@ Descriptor.__set_name__{args!r}')

    def __set__(self, instance, value):  
        args = (self, instance, value)
        print(f'@ Descriptor.__set__{args!r}')

    def __repr__(self):
        return '<Descriptor instance>'


print('@ builderlib module end')

>>> import builderlib
@ builderlib module start
@ Builder body
@ Descriptor body
@ builderlib module end

#!/usr/bin/env python3

from builderlib import Builder, deco, Descriptor

print('# evaldemo module start')

@deco  
class Klass(Builder):  
    print('# Klass body')

    attr = Descriptor()  

    def __init__(self):
        super().__init__()
        print(f'# Klass.__init__({self!r})')

    def __repr__(self):
        return '<Klass instance>'


def main():  
    obj = Klass()
    obj.method_a()
    obj.method_b()
    obj.attr = 999

if __name__ == '__main__':
    main()

print('# evaldemo module end')

>>> import evaldemo
@ builderlib module start  
@ Builder body
@ Descriptor body
@ builderlib module end
# evaldemo module start
# Klass body  
@ Descriptor.__init__(<Descriptor instance>)  
@ Descriptor.__set_name__(<Descriptor instance>,
      <class 'evaldemo.Klass'>, 'attr')                
@ Builder.__init_subclass__(<class 'evaldemo.Klass'>)  
@ deco(<class 'evaldemo.Klass'>)  
# evaldemo module end

$ ./evaldemo.py
[... 9 lines omitted ...]
@ deco(<class '__main__.Klass'>)  
@ Builder.__init__(<Klass instance>)  
# Klass.__init__(<Klass instance>)
@ SuperA.__init_subclass__:inner_0(<Klass instance>)  
@ deco:inner_1(<Klass instance>)  
@ Descriptor.__set__(<Descriptor instance>, <Klass instance>, 999)  
# evaldemo module end

[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).

Tim Peters, inventor of the Timsort algorithm and prolific Python contributor13

A metaclass is a class factory. In contrast with record_factory from Example 24-2, a metaclass is written as a class. In other words, a metaclass is a class whose instances are classes. Figure 24-1 depicts a metaclass using the Mills & Gizmos Notation: a mill producing another mill.

Figure 24-1. A metaclass is a class that builds classes.

Consider the Python object model: classes are objects, therefore each class must be an instance of some other class. By default, Python classes are instances of type. In other words, type is the metaclass for most built-in and user-defined classes:

>>> str.__class__
<class 'type'>
>>> from bulkfood_v5 import LineItem
>>> LineItem.__class__
<class 'type'>
>>> type.__class__
<class 'type'>

To avoid infinite regress, the class of type is type, as the last line shows.

Note that I am not saying that str or LineItem are subclasses of type. What I am saying is that str and LineItem are instances of type. They all are subclasses of object. Figure 24-2 may help you confront this strange reality.

Figure 24-2. Both diagrams are true. The left one emphasizes that str, type, and LineItem are subclasses of object. The right one makes it clear that str, object, and LineItem are instances type, because they are all classes.

The next snippet shows that the class of collections.Iterable is abc.ABCMeta. Note that Iterable is an abstract class, but ABCMeta is a concrete class—after all, Iterable is an instance of ABCMeta:

>>> from collections.abc import Iterable
>>> Iterable.__class__
<class 'abc.ABCMeta'>
>>> import abc
>>> from abc import ABCMeta
>>> ABCMeta.__class__
<class 'type'>

Ultimately, the class of ABCMeta is also type. Every class is an instance of type, directly or indirectly, but only metaclasses are also subclasses of type. That’s the most important relationship to understand metaclasses: a metaclass, such as ABCMeta, inherits from type the power to construct classes. Figure 24-3 illustrates this crucial relationship.

Figure 24-3. Iterable is a subclass of object and an instance of ABCMeta. Both object and ABCMeta are instances of type, but the key relationship here is that ABCMeta is also a subclass of type, because ABCMeta is a metaclass. In this diagram, Iterable is the only abstract class.

The important takeaway here is that metaclasses are subclasses of type, and that’s what makes them work as class factories. A metaclass can customize its instances by implementing special methods, as the next sections demonstrate.

class Klass(SuperKlass, metaclass=MetaKlass):
    x = 42
    def __init__(self, y):
        self.y = y

    >>> class Point(Bunch):
    ...     x = 0.0
    ...     y = 0.0
    ...     color = 'gray'
    ...
    >>> Point(x=1.2, y=3, color='green')
    Point(x=1.2, y=3, color='green')
    >>> p = Point()
    >>> p.x, p.y, p.color
    (0.0, 0.0, 'gray')
    >>> p
    Point()

    >>> Point(x=1, y=2, z=3)
    Traceback (most recent call last):
      ...
    AttributeError: No slots left for: 'z'
    >>> p = Point(x=21)
    >>> p.y = 42
    >>> p
    Point(x=21, y=42)
    >>> p.flavor = 'banana'
    Traceback (most recent call last):
      ...
    AttributeError: 'Point' object has no attribute 'flavor'

class MetaBunch(type):  
    def __new__(meta_cls, cls_name, bases, cls_dict):  

        defaults = {}  

        def __init__(self, **kwargs):  
            for name, default in defaults.items():  
                setattr(self, name, kwargs.pop(name, default))
            if kwargs:  
                extra = ', '.join(kwargs)
                raise AttributeError(f'No slots left for: {extra!r}')

        def __repr__(self):  
            rep = ', '.join(f'{name}={value!r}'
                            for name, default in defaults.items()
                            if (value := getattr(self, name)) != default)
            return f'{cls_name}({rep})'

        new_dict = dict(__slots__=[], __init__=__init__, __repr__=__repr__)  

        for name, value in cls_dict.items():  
            if name.startswith('__') and name.endswith('__'):  
                if name in new_dict:
                    raise AttributeError(f"Can't set {name!r} in {cls_name!r}")
                new_dict[name] = value
            else:  
                new_dict['__slots__'].append(name)
                defaults[name] = value
        return super().__new__(meta_cls, cls_name, bases, new_dict)  


class Bunch(metaclass=MetaBunch):  
    pass

#!/usr/bin/env python3

from builderlib import Builder, deco, Descriptor
from metalib import MetaKlass  

print('# evaldemo_meta module start')

@deco
class Klass(Builder, metaclass=MetaKlass):  
    print('# Klass body')

    attr = Descriptor()

    def __init__(self):
        super().__init__()
        print(f'# Klass.__init__({self!r})')

    def __repr__(self):
        return '<Klass instance>'


def main():
    obj = Klass()
    obj.method_a()
    obj.method_b()
    obj.method_c()  
    obj.attr = 999


if __name__ == '__main__':
    main()

print('# evaldemo_meta module end')

print('% metalib module start')

import collections

class NosyDict(collections.UserDict):
    def __setitem__(self, key, value):
        args = (self, key, value)
        print(f'% NosyDict.__setitem__{args!r}')
        super().__setitem__(key, value)

    def __repr__(self):
        return '<NosyDict instance>'

class MetaKlass(type):
    print('% MetaKlass body')

    @classmethod  
    def __prepare__(meta_cls, cls_name, bases):  
        args = (meta_cls, cls_name, bases)
        print(f'% MetaKlass.__prepare__{args!r}')
        return NosyDict()  

    def __new__(meta_cls, cls_name, bases, cls_dict):  
        args = (meta_cls, cls_name, bases, cls_dict)
        print(f'% MetaKlass.__new__{args!r}')
        def inner_2(self):
            print(f'% MetaKlass.__new__:inner_2({self!r})')

        cls = super().__new__(meta_cls, cls_name, bases, cls_dict.data)  

        cls.method_c = inner_2  

        return cls  

    def __repr__(cls):  
        cls_name = cls.__name__
        return f"<class {cls_name!r} built by MetaKlass>"

print('% metalib module end')

>>> import metalib
% metalib module start
% MetaKlass body
% metalib module end

>>> import evaldemo_meta
@ builderlib module start
@ Builder body
@ Descriptor body
@ builderlib module end
% metalib module start
% MetaKlass body
% metalib module end
# evaldemo_meta module start  
% MetaKlass.__prepare__(<class 'metalib.MetaKlass'>, 'Klass',  
                        (<class 'builderlib.Builder'>,))
% NosyDict.__setitem__(<NosyDict instance>, '__module__', 'evaldemo_meta')  
% NosyDict.__setitem__(<NosyDict instance>, '__qualname__', 'Klass')
# Klass body
@ Descriptor.__init__(<Descriptor instance>)  
% NosyDict.__setitem__(<NosyDict instance>, 'attr', <Descriptor instance>)  
% NosyDict.__setitem__(<NosyDict instance>, '__init__',
                       <function Klass.__init__ at …>)  
% NosyDict.__setitem__(<NosyDict instance>, '__repr__',
                       <function Klass.__repr__ at …>)
% NosyDict.__setitem__(<NosyDict instance>, '__classcell__', <cell at …: empty>)
% MetaKlass.__new__(<class 'metalib.MetaKlass'>, 'Klass',
                    (<class 'builderlib.Builder'>,), <NosyDict instance>)  
@ Descriptor.__set_name__(<Descriptor instance>,
                          <class 'Klass' built by MetaKlass>, 'attr')  
@ Builder.__init_subclass__(<class 'Klass' built by MetaKlass>)
@ deco(<class 'Klass' built by MetaKlass>)
# evaldemo_meta module end

$ ./evaldemo_meta.py
[... 20 lines omitted ...]
@ deco(<class 'Klass' built by MetaKlass>)  
@ Builder.__init__(<Klass instance>)
# Klass.__init__(<Klass instance>)
@ SuperA.__init_subclass__:inner_0(<Klass instance>)
@ deco:inner_1(<Klass instance>)
% MetaKlass.__new__:inner_2(<Klass instance>)  
@ Descriptor.__set__(<Descriptor instance>, <Klass instance>, 999)
# evaldemo_meta module end

I don’t want to encourage premature optimization and overengineering, so here is a make-believe scenario to justify rewriting checkedlib.py with __slots__, which requires the application of a metaclass. Feel free to skip it.

from checkedlib import Checked

class Movie(Checked):
    title: str
    year: int
    box_office: float

if __name__ == '__main__':
    movie = Movie(title='The Godfather', year=1972, box_office=137)
    print(movie)
    print(movie.title)

Consider Figure 24-4. The Mills & Gizmos Notation complements the UML class diagram by making the relationship between classes and instances more visible.

Now let’s study the code, starting with the Field class, shown in Example 24-21.

The Field descriptor class is now a bit different. In the previous examples, each Field descriptor instance stored its value in the managed instance using an attribute of the same name. For example, in the Movie class, the title descriptor stored the field value in a title attribute in the managed instance. This made it unnecessary for Field to provide a __get__ method.

However, when a class like Movie uses __slots__, it cannot have class attributes and instance attributes with the same name. Each descriptor instance is a class attribute, and now we need separate per-instance storage attributes. The code uses the descriptor name prefixed with a single _. Therefore Field instances have separate name and storage_name attributes, and we implement Field.__get__.

Figure 24-4. UML class diagram annotated with MGN: the CheckedMeta meta-mill builds the Movie mill. The Field mill builds the title, year, and box_office descriptors, which are class attributes of Movie. The per-instance data for the fields is stored in the _title, _year, and _box_office instance attributes of Movie. Note the package boundary of checkedlib. The developer of Movie doesn’t need to grok all the machinery inside checkedlib.py.

Example 24-21 shows the source code for Field, with callouts describing only the changes in this version.

class Field:
    def __init__(self, name: str, constructor: Callable) -> None:
        if not callable(constructor) or constructor is type(None):
            raise TypeError(f'{name!r} type hint must be callable')
        self.name = name
        self.storage_name = '_' + name  
        self.constructor = constructor

    def __get__(self, instance, owner=None):
        if instance is None:  
            return self
        return getattr(instance, self.storage_name)  

    def __set__(self, instance: Any, value: Any) -> None:
        if value is ...:
            value = self.constructor()
        else:
            try:
                value = self.constructor(value)
            except (TypeError, ValueError) as e:
                type_name = self.constructor.__name__
                msg = f'{value!r} is not compatible with {self.name}:{type_name}'
                raise TypeError(msg) from e
        setattr(instance, self.storage_name, value)  

Compute storage_name from the name argument.

If __get__ gets None as the instance argument, the descriptor is being read from the managed class itself, not a managed instance. So we return the descriptor.

Otherwise, return the value stored in the attribute named storage_name.

__set__ now uses setattr to set or update the managed attribute.

Example 24-22 shows the code for the metaclass that drives this example.

class CheckedMeta(type):

    def __new__(meta_cls, cls_name, bases, cls_dict):  
        if '__slots__' not in cls_dict:  
            slots = []
            type_hints = cls_dict.get('__annotations__', {})  
            for name, constructor in type_hints.items():   
                field = Field(name, constructor)  
                cls_dict[name] = field  
                slots.append(field.storage_name)  

            cls_dict['__slots__'] = slots  

        return super().__new__(
                meta_cls, cls_name, bases, cls_dict)  

__new__ is the only method implemented in CheckedMeta.

Only enhance the class if its cls_dict doesn’t include __slots__. If __slots__ is already present, assume it is the Checked base class and not a user-defined subclass, and build the class as is.

To get the type hints in prior examples, we used typing.get_type_hints, but that requires an existing class as the first argument. At this point, the class we are configuring does not exist yet, so we need to retrieve the __annotations__ directly from the cls_dict—the namespace of the class under construction, which Python passes as the last argument to the metaclass __new__.

Iterate over type_hints to…

…build a Field for each annotated attribute…

…overwrite the corresponding entry in cls_dict with the Field instance…

…and append the storage_name of the field in the list we’ll use to…

…populate the __slots__ entry in cls_dict—the namespace of the class under construction.

Finally, we call super().__new__.

The last part of metaclass/checkedlib.py is the Checked base class that users of this library will subclass to enhance their classes, like Movie.

The code for this version of Checked is the same as Checked in initsub/checkedlib.py (listed in Example 24-5 and Example 24-6), with three changes:

1. Added an empty __slots__ to signal to CheckedMeta.__new__ that this class doesn’t require special processing.

2. Removed __init_subclass__. Its job is now done by CheckedMeta.__new__.

3. Removed __setattr__. It became redundant because adding __slots__ to the user-defined class prevents setting undeclared attributes.

Example 24-23 is a complete listing of the final version of Checked.

class Checked(metaclass=CheckedMeta):
    __slots__ = ()  # skip CheckedMeta.__new__ processing

    @classmethod
    def _fields(cls) -> dict[str, type]:
        return get_type_hints(cls)

    def __init__(self, **kwargs: Any) -> None:
        for name in self._fields():
            value = kwargs.pop(name, ...)
            setattr(self, name, value)
        if kwargs:
            self.__flag_unknown_attrs(*kwargs)

    def __flag_unknown_attrs(self, *names: str) -> NoReturn:
        plural = 's' if len(names) > 1 else ''
        extra = ', '.join(f'{name!r}' for name in names)
        cls_name = repr(self.__class__.__name__)
        raise AttributeError(f'{cls_name} object has no attribute{plural} {extra}')

    def _asdict(self) -> dict[str, Any]:
        return {
            name: getattr(self, name)
            for name, attr in self.__class__.__dict__.items()
            if isinstance(attr, Field)
        }

    def __repr__(self) -> str:
        kwargs = ', '.join(
            f'{key}={value!r}' for key, value in self._asdict().items()
        )
        return f'{self.__class__.__name__}({kwargs})'

This concludes the third rendering of a class builder with validated descriptors.

The next section covers some general issues related to metaclasses.

Metaclasses are powerful, but tricky. Before deciding to implement a metaclass, consider the following points.

TypeError: metaclass conflict: the metaclass of a derived class
must be a (non-strict) subclass of the metaclasses of all its bases

class Record(abc.ABC, metaclass=PersistentMeta):
    pass

When I updated this chapter for the second edition, I needed to find simple but illuminating examples to replace the bulkfood LineItem code that no longer require metaclasses since Python 3.6.

    >>> class Flavor(AutoConst):
    ...     banana
    ...     coconut
    ...     vanilla
    ...
    >>> Flavor.vanilla
    2
    >>> Flavor.banana, Flavor.coconut
    (0, 1)

class AutoConstMeta(type):
    def __prepare__(name, bases, **kwargs):
        return WilyDict()

class AutoConst(metaclass=AutoConstMeta):
    pass

When Python processes the namespace of the user’s class and reads banana, it looks up that name in the mapping provided by __prepare__: an instance of WilyDict. WilyDict implements __missing__, covered in “The __missing__ Method”. The WilyDict instance initially has no 'banana' key, so the __missing__ method is triggered. It makes an item on the fly with the key 'banana' and the value 0, returning that value. Python is happy with that, then tries to retrieve 'coconut'. WilyDict promptly adds that entry with the value 1, returning it. The same happens with 'vanilla', which is then mapped to 2.

We’ve seen __prepare__ and __missing__ before. The real innovation is how JS put them together.

Here is the source code for WilyDict, also from autoconst.py:

class WilyDict(dict):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.__next_value = 0

    def __missing__(self, key):
        if key.startswith('__') and key.endswith('__'):
            raise KeyError(key)
        self[key] = value = self.__next_value
        self.__next_value += 1
        return value

While experimenting, I found that Python looked up __name__ in the namespace of the class under construction, causing WilyDict to add a __name__ entry, and increment __next_value. So I added that if statement in __missing__ to raise KeyError for keys that look like dunder attributes.

The autoconst.py package both requires and illustrates mastery of Python’s dynamic class building machinery.

I had a great time adding more functionality to AutoConstMeta and AutoConst, but instead of sharing my experiments, I will let you have fun playing with JS’s ingenious hack.

Here are some ideas:

• Make it possible to retrieve the constant name if you have the value. For example, Flavor[2] could return 'vanilla'. You can to this by implementing __getitem__ in AutoConstMeta. Since Python 3.9, you can implement __class_getitem__ in AutoConst itself.

• Support iteration over the class, by implementing __iter__ on the metaclass. I would make the __iter__ yield the constants as (name, value) pairs.

• Implement a new Enum variant. This would be a major undertaking, because the enum package is full of tricks, including the EnumMeta metaclass with hundreds of lines of code and a nontrivial __prepare__ method.

Enjoy!

Metaclasses, as well as class decorators and __init_subclass__ are useful for:

• Subclass registration

• Subclass structural validation

• Applying decorators to many methods at once

• Object serialization

• Object-relational mapping

• Object-based persistence

• Implementing special methods at the class level

• Implementing class features found in other languages, such as traits and aspect-oriented programming

Class metaprogramming can also help with performance issues in some cases, by performing tasks at import time that otherwise would execute repeatedly at runtime.

To wrap up, let’s recall Alex Martelli’s final advice from his essay “Waterfowl and ABCs”:

And, don’t define custom ABCs (or metaclasses) in production code… if you feel the urge to do so, I’d bet it’s likely to be a case of “all problems look like a nail”-syndrome for somebody who just got a shiny new hammer—you (and future maintainers of your code) will be much happier sticking with straightforward and simple code, eschewing such depths.

I believe Martelli’s advice applies not only to ABCs and metaclasses, but also to class hierarchies, operator overloading, function decorators, descriptors, class decorators, and class builders using __init_subclass__.

Those powerful tools exist primarily to support library and framework development. Applications naturally should use those tools, as provided by the Python standard library or external packages. But implementing them in application code is often premature abstraction.

Good frameworks are extracted, not invented.¹⁸

David Heinemeier Hansson, creator of Ruby on Rails

Chapter Summary

This chapter started with an overview of attributes found in class objects, such as __qualname__ and the __subclasses__() method. Next, we saw how the type built-in can be used to construct classes at runtime.

Caleb Hattingh—a technical reviewer of this book—wrote the autoslot package, providing a metaclass to automatically create a __slots__ attribute in a user-defined class by inspecting the bytecode of __init__ and finding all assignments to attributes of self. It’s useful and also an excellent example to study: only 74 lines of code in autoslot.py, including 20 lines of comments explaining the most difficult parts.

The essential references for this chapter in the Python documentation are “3.3.3. Customizing class creation” in the “Data Model” chapter of The Python Language Reference, which covers __init_subclass__ and metaclasses. The type class documentation in the “Built-in Functions” page, and “4.13. Special Attributes” of the “Built-in Types” chapter in the The Python Standard Library are also essential reading.

In the The Python Standard Library, the types module documentation covers two functions added in Python 3.3 that simplify class metaprogramming: types.new_class and types.prepare_class.

Class decorators were formalized in PEP 3129—Class Decorators, written by Collin Winter, with the reference implementation authored by Jack Diederich. The PyCon 2009 talk “Class Decorators: Radically Simple” (video), also by Jack Diederich, is a quick introduction to the feature. Besides @dataclass, an interesting—and much simpler—example of a class decorator in Python’s standard library is functools.total_ordering that generates special methods for object comparison.

For metaclasses, the main reference in Python’s documentation is PEP 3115—Metaclasses in Python 3000, in which the __prepare__ special method was introduced.

Python in a Nutshell, 3rd ed., by Alex Martelli, Anna Ravenscroft, and Steve Holden, is authoritative, but was written before PEP 487—Simpler customization of class creation came out. The main metaclass example in that book—MetaBunch—is still valid, because it can’t be written with simpler mechanisms. Brett Slatkin’s Effective Python, 2nd ed. (Addison-Wesley) has several up-to-date examples of class building techniques, including metaclasses.

To learn about the origins of class metaprogramming in Python, I recommend Guido van Rossum’s paper from 2003, “Unifying types and classes in Python 2.2”. The text applies to modern Python as well, as it covers what were then called the “new-style” class semantics—the default semantics in Python 3—including descriptors and metaclasses. One of the references cited by Guido is Putting Metaclasses to Work: a New Dimension in Object-Oriented Programming, by Ira R. Forman and Scott H. Danforth (Addison-Wesley), a book to which he gave five stars on Amazon.com, adding the following review:

This book contributed to the design for metaclasses in Python 2.2

Too bad this is out of print; I keep referring to it as the best tutorial I know for the difficult subject of cooperative multiple inheritance, supported by Python via the super() function.¹⁹

If you are keen on metaprogramming, you may wish Python had the ultimate metaprogramming feature: syntactic macros, as offered in the Lisp family of languages and—more recently—by Elixir and Rust. Syntactic macros are more powerful and less error prone than the primitive code substitution macros in the C language. They are special functions that rewrite source code using custom syntax into standard code before the compilation step, enabling developers to introduce new language constructs without changing the compiler. Like operator overloading, syntactic macros can be abused. But as long as the community understands and manages the downsides, they support powerful and user-friendly abstractions, like DSLs (Domain-Specific Languages). In September 2020, Python core developer Mark Shannon posted PEP 638—Syntactic Macros, advocating just that. A year after it was initially published, PEP 638 was still in draft and there were no ongoing discussions about it. Clearly it’s not a top priority for the Python core developers. I would like to see PEP 638 further discussed and eventually approved. Syntactic macros would allow the Python community to experiment with controversial new features, such as the walrus operator (PEP 572), pattern matching (PEP 634), and alternative rules for evaluating type hints (PEPs 563 and 649) before making permanent changes to the core language. Meanwhile, you can get a taste of syntactic macros with the MacroPy package.