Extending Types by Embedding

Do you remember those set functions we wrote in Chapters 16 and 18? Here’s what they look like brought back to life as a Python class. Example 32-1 (file setwrapper.py) implements a new set object type by moving set functions to methods and adding some basic operator overloading. For the most part, this class just wraps a Python list with extra set operations. But because it’s a class, it also supports multiple instances and customization by inheritance in subclasses. Unlike our earlier functions, using classes here allows us to make multiple self-contained set objects with preset data and behavior rather than passing lists into functions manually.

class Set:
   def __init__(self, value = []):    # Constructor
       self.data = []                 # Manages a list
       self.concat(value)             # Removes duplicates

   def intersect(self, other):        # other is any iterable
       res = []                       # self is the subject
       for x in self.data:
           if x in other:             # Pick common items
               res.append(x)
       return Set(res)                # Return a new Set

   def union(self, other):            # other is any iterable
       res = self.data[:]             # Copy of my list
       for x in other:                # Add items in other
           if not x in res:
               res.append(x)
       return Set(res)

   def concat(self, value):           # value: list, Set...
       for x in value:                # Removes duplicates
          if not x in self.data:
               self.data.append(x)

   def __len__(self):          return len(self.data)            # len(self), if self
   def __getitem__(self, key): return self.data[key]            # self[i], self[i:j]
   def __and__(self, other):   return self.intersect(other)     # self & other
   def __or__(self, other):    return self.union(other)         # self | other
   def __repr__(self):         return f'Set({self.data!r})'     # print(self),...
   def __iter__(self):         return iter(self.data)           # for x in self,...

To use this class, we make instances, call methods, and run defined operators as usual:

$ python3
>>> from setwrapper import Set
>>> x = Set([1, 3, 5, 7, 3])
>>> x.union(Set([1, 4, 7]))
Set([1, 3, 5, 7, 4])
>>> x | Set([1, 4, 6, 4])
Set([1, 3, 5, 7, 4, 6])

Overloading operations such as indexing and iteration also enables instances of our Set class to often masquerade as real lists. Because you will interact with and extend this class in an exercise at the end of this chapter, we’ll put this code on the back burner until its solution in Appendix B.

Extending Types by Subclassing

While the prior section’s embedding works, Python’s built-in types can also be subclassed directly. In fact, type-conversion functions such as list, str, dict, and tuple are really built-in type names; although transparent to your script, a type-conversion call (e.g., list('text')) is really an invocation of a type’s constructor.

This allows you to customize or extend the behavior of built-in types with user-defined class statements: simply subclass the type names to customize them. Instances of your type subclasses can generally be used anywhere that the original built-in type can appear. For example, suppose you have trouble getting used to the fact that Python list offsets begin at 0 instead of 1. Not to worry—you can always code your own subclass that customizes this core behavior of lists, and Example 32-2 shows how.

"""
Subclass built-in list type/class.
Map 1..N to 0..N-1, call back to built-in version.
"""

class MyList(list):
    def __getitem__(self, offset):
        print(f'<indexing {self} at {offset}>')
        return list.__getitem__(self, offset - 1)

if __name__ == '__main__':
    print(list('abc'))
    x = MyList('abc')               # __init__ inherited from list
    print(x)                        # __str__/__repr__ inherited from list

    print(x[1])                     # MyList.__getitem__
    print(x[3])                     # Customizes list superclass method

    x.append('hack!'); print(x)     # Attributes from list superclass
    x.reverse();       print(x)

In this file, the MyList subclass extends the built-in list’s __getitem__ indexing method only, to map indexes 1 to N back to the required 0 to N−1. Really, all it does is decrement the submitted index and call back to the superclass’s version of indexing, but it’s enough to do the trick:

$ python3 typesubclass.py
['a', 'b', 'c']
['a', 'b', 'c']
<indexing ['a', 'b', 'c'] at 1>
a
<indexing ['a', 'b', 'c'] at 3>
c
['a', 'b', 'c', 'hack!']
['hack!', 'c', 'b', 'a']

This output also includes tracing text the class prints on indexing. Of course, whether changing indexing this way is a good idea, in general, is another issue—users of your MyList class may very well be confused by such a core departure from Python sequence behavior. The ability to customize built-in types this way can be a powerful asset, though.

For instance, this coding pattern gives rise to an alternative way to code a set—as a subclass of the built-in list type rather than a standalone class that manages an embedded list object, as shown in the prior section. As discussed in Chapter 5, Python today comes with a powerful built-in set object, along with literal and comprehension syntax for making new sets. Coding one yourself, though, is still a great way to learn about type subclassing in general.

The code in Example 32-3, file setsubclass.py, customizes lists to add just methods and operators related to set processing. Because all other behavior is inherited from the built-in list superclass, this makes for a shorter and simpler alternative—everything not defined here is routed to list directly.

class Set(list):
    def __init__(self, value = []):      # Constructor
        list.__init__(self)              # Customizes list
        self.concat(value)               # Copies mutable defaults

    def intersect(self, other):          # other is any iterable
        res = []                         # self is the subject
        for x in self:
            if x in other:               # Pick common items
                res.append(x)
        return Set(res)                  # Return a new Set

    def union(self, other):              # other is any iterable
        res = Set(self)                  # Copy me and my list
        res.concat(other)
        return res

    def concat(self, value):             # value: list, Set, etc.
        for x in value:                  # Removes duplicates
            if not x in self:
                self.append(x)

    def __and__(self, other): return self.intersect(other)
    def __or__(self, other):  return self.union(other)
    def __repr__(self):       return f'Set({list.__repr__(self)})'

if __name__ == '__main__':
    x = Set([1, 3, 5, 7])
    y = Set([2, 1, 4, 5, 6])
    print(x, y, len(x))
    print(x.intersect(y), y.union(x))
    print(x & y, x | y)
    x.reverse(); print(x)

Here is the output of the self-test code at the end of this file. Because subclassing core types is a somewhat advanced feature with a limited audience, we’ll end this topic here, but you’re invited to trace through these results in the code to study its behavior:

$ python3 setsubclass.py
Set([1, 3, 5, 7]) Set([2, 1, 4, 5, 6]) 4
Set([1, 5]) Set([2, 1, 4, 5, 6, 3, 7])
Set([1, 5]) Set([1, 3, 5, 7, 2, 4, 6])
Set([7, 5, 3, 1])

Subtleties: some inherited list operations may introduce duplicates to our Set, and there are more efficient ways to implement sets with dictionaries in Python, which replace the nested linear search scans in the set implementations shown here with more direct dictionary index operations (hashing) and so run much quicker. If you’re interested in sets, also take another look at the set object type we explored in Chapter 5; this type provides extensive set operations as built-in tools. Set implementations are fun to experiment with but not strictly required in Python today.

More important here is the question of why we can subclass built-in types like list at all. The next section solves the mystery—at least as much as this chapter can.

Classes are Types are Classes

While you probably shouldn’t ponder the preceding definitions before operating heavy machinery, it’s easy to see all this in code. The type built-in with one argument returns any object’s type, which is normally the same as the object’s __class__, and isinstance checks whether an object inherits from another. Here’s the story for user-defined classes (a.k.a. types):

>>> class Hack: pass         # A humble user-defined class
>>> I = Hack()               # Make an instance by calling the class

>>> type(I)                  # Type is the user-defined class of origin
<class '__main__.Hack'>

>>> type(Hack)               # User-defined classes are instances of type
<class 'type'>

>>> I.__class__, C.__class__
(<class '__main__.Hack'>, <class 'type'>)

>>> isinstance(I, object), isinstance(Hack, object)
(True, True)

This works the same for built-in types (a.k.a. classes), but there is also literal syntax for generating instances:

>>> I = 'hack'               # Make an instance by literal syntax, or str()

>>> type(I)                  # Built-in objects are instances of classes
<class 'str'>

>>> type(str)                # Built-in classes are instances of type
<class 'type'>

>>> I.__class__, str.__class__
(<class 'str'>, <class 'type'>)

>>> isinstance(I, object), isinstance(str, object)
(True, True)

In fact, type itself reports in as a class, though it has no type but itself—circularly capping the chain:

>>> type(type)                 # The type class ends the chain
<class 'type'>
>>> type(type(type))           # Hmm... the top of the chain
<class 'type'>

This model may seem academic (and to some extent is), but it allows us to specialize built-in types with normal user-defined classes and bears on type-testing code: you must know what a type is to test it accurately.

Some Instances Are More Equal than Others

It’s tempting to simply take away from the foregoing that classes and types are the same, but this story is richer than that may imply: the instances we make from these objects diverge in both functionality and inheritance.

To truly understand how, we have to briefly factor in metaclasses—classes that generate other classes. The type built-in itself is a metaclass and may be customized with user-defined subclasses. These subclasses are coded with normal class statements, selected with special syntax in class headers, and designed to play metaclass roles, but they won’t be covered in full until Chapter 40.

In brief, though, the complete relationship between instances, classes, and types is as follows:

• Instances are created from classes—both built-in and user-defined

• Classes themselves are created from the built-in type class or one of its subclasses

• The object built-in class is a superclass to every object—instance, class, or both

Although everything is ultimately an “instance” in Python, there are two fundamentally different kinds of instances, and conflating these only serves to mask the true complexity of the model. It doesn’t help to distinguish these as instances of user-defined classes or not: metaclasses may be user-defined classes too. Nor is this about being a subclass of a type: the real fork in this model is that classes are created from a type class specially.

As you’ll learn in full later, classes define their types with optional metaclass syntax that defaults to type if omitted, but other instances define their types by the class calls or literal syntax we’ve used so far:

class C(metaclass=Meta): …     # Class creation: metaclass defaults to "type"

I = C(…)                       # Instance creation: user-defined classes
X = [1, 2]                     # Instance creation: built-in classes

While both produce instances in some sense, these different syntaxes create very different kinds of instances—class and nonclass—with fundamentally different behaviors:

Nonclass instances do not make instances

Classes are created from type (or other metaclass), similar to the way instances are created from classes. Once created, though, the analogy fails: classes create instances of their own, but nonclass instances do not.

Classes have an extra inheritance search

There are really two inheritance trees and searches in Python, which are distinct but not entirely disjoint. The secondary tree is formed by type and its subclasses and is searched only for classes, not nonclass instances.

In other words, nonclass instances seal off the instantiation chain, and inheritance differs for nonclass instances and classes themselves—even though the latter are also instances of type. All of this boils down to different creation syntax that makes different kinds of objects, which are often confusingly lumped together as “instances”:

>>> class C: pass                                # A type instance
>>> I = C()                                      # A nonclass instance
>>> isinstance(I, type), isinstance(C, type)     # Only classes are types
(False, True)

While types and classes may be synonymous, the instances we create from them vary per creation code.

The Inheritance Bifurcation

Though we can’t get into full details here, the inheritance search used for classes (a.k.a. types) differs from what we’ve seen so far and may be their most profound distinction. In short, inheritance is always based on the MRO (method resolution order) we studied in the last chapter’s “Multiple Inheritance and the MRO”, but varies as follows:

Nonclass inheritance

As we’ve seen, inheritance run on a nonclass instance searches the __dict__ attributes of instance, class, and superclasses, per the MRO order we studied in the prior chapter. This works by first checking the instance, then following the instance’s __class__ to its class, and finally following each class’s __bases__ to superclasses. Technically, __bases__ are used to make an __mro__ at __class__, which inheritance scans.

Class inheritance

Inheritance run on a class directly, though, first searches the __dict__ of the class and all its supers available from __bases__ as usual, but then also searches the separate class tree formed by the type class and its metaclass subclasses. The second part of this works by following the class’s own __class__ to its type class tree and using __bases__ and MROs there, too—but only as a last resort and only for inheritance run on classes.

In fact, if you know where to look, you can inspect the inheritance sources that differ for nonclass instances like I and classes like C in the prior example—though the underscores and displays aren’t pretty:

>>> isinstance(I, C), type(I)
(True, <class '__main__.C'>)

>>> I, I.__class__, I.__class__.__bases__
(<__main__.C object at 0x101265d60>, <class '__main__.C'>, (<class 'object'>,))

>>> C, C.__bases__, C.__class__, C.__class__.__bases__
(<class '__main__.C'>, (<class 'object'>,), <class 'type'>, (<class 'object'>,))

But due to the way MROs are computed from __bases__ and scanned, it’s more accurate to think of inheritance’s different search orders for nonclass instances and classes as follows—where each __mro__ is a flattened tree:

>>> I, I.__class__.__mro__
(<__main__.C object at 0x101265d60>, (<class '__main__.C'>, <class 'object'>))

>>> C.__mro__, C.__class__.__mro__
((<class '__main__.C'>, <class 'object'>), (<class 'type'>, <class 'object'>))

And because each item’s __dict__ is checked, the ordered set of candidates searched by inheritance for nonclass instances I and classes C is ultimately and respectively as follows—with two MRO scans of flattened trees for classes only, and ignoring the fact that some kinds of descriptors, introduced ahead, take precedence in both trees as you’ll learn in Chapter 40:

[I.__dict__] + [x.__dict__ for x in I.__class__.__mro__]

[x.__dict__ for x in C.__mro__] + [x.__dict__ for x in C.__class__.__mro__]

Wait—there’s a second tree in inheritance? Well, yes, though it doesn’t come into play in the vast majority of application code. The type/metaclass tree is used in advanced class-management roles and, even then, is often limited to class customization at class creation time.

Still, this secondary tree, along with the descriptors’ special cases omitted here, bifurcates and convolutes the inheritance story, especially compared to its prior forms. It also explains why some class attributes like __bases__ are not inherited by nonclass instances—they’re located in the secondary tree (i.e., MRO) searched only for classes:

>>> '__bases__' in I.__dict__                # Not in instance
False
>>> '__bases__' in C.__dict__                # Not in instance's class
False
>>> '__bases__' in C.__class__.__dict__      # In instance's class's class
True

Because of the two-tree inheritance model, such names inherited by classes are not inherited by their instances:

>>> C.__bases__                              # Instance does not inherit!
(<class 'object'>,)
>>> I.__bases__
AttributeError: 'C' object has no attribute '__bases__'. Did you mean: '__class__'?

The Metaclass/Class Dichotomy

So, where does this odd tale of type/class unification leave us? Types indeed behave as classes, and this allows us to extend them with normal class syntax in both the primary and metaclass trees. But it also comes with noticeable seams, including special-case syntax for class instantiation, an extra type-tree search for classes only, two very different kinds of instances, and unique semantics for metaclasses that customize types (a.k.a. classes), which we’ll uncover later.

In fact, a reasonable argument can be made that the type/class dichotomy of earlier Pythons may simply have morphed into one of metaclass/class—which trades a straightforward distinction for all the seams just enumerated and muddles inheritance and the fundamental meaning of names in Python everywhere to support what in the end is a very rare use case. As usual, the net merit of the morph is yours to weigh.

To be fair, some of the widespread confusion this model has spawned may stem from type itself: it’s overloaded to either return a sole argument’s type, or generate a new instance of itself for multiple arguments—just like other constructors and equivalent to what a class statement does to make a class object:

type(object)                                     # Fetch object type
type(classname, superclasses, attributedict)     # Make a class/type

The first of these roles might have been better named “typeof,” but the second will have to await the metaclass preview later in this chapter and the extended coverage in Chapter 40.

And One “object” to Rule Them All

To round out this topic, keep in mind that because topmost classes inherit from the built-in class object, every object derives (i.e., inherits) from it, whether directly or through a superclass—and whether you code object or not:

>>> class C: pass
>>> class D(object): pass

>>> dir(C) == dir(D)
True
>>> C.__bases__, D.__bases__
((<class 'object'>,), (<class 'object'>,))

In fact, the type class inherits from the object class, and object inherits from type, even though the two are different objects—a circular relationship that crowns the object model and may make your cranium catch fire (to avoid combustion, keep in mind that isinstance is true for either a subclass relationship or creation source, though the latter may also imply inheritance through the secondary type-class tree for classes):

>>> type is object
False
>>> type(type), type(object)
(<class 'type'>, <class 'type'>)
>>> isinstance(type, object), isinstance(object, type)
(True, True)
>>> type.__bases__, object.__bases__
((<class 'object'>,), ())

Strange though it may seem, this has a number of practical consequences. For one thing, it means that we sometimes must be aware of the method defaults that come with the implicit (or explicit) object root class. As we noted in earlier chapters, for instance, the object class comes with a __repr__ for display:

>>> class C: pass                     # All classes inherit object defaults
>>> X = C()
>>> X.__repr__
<method-wrapper '__repr__' of C object at 0x1091a7920>

For another, this also allows us to write code that can safely assume and use an object superclass. As an example, we can rely on it to be a call-chain “anchor” in some super built-in roles described ahead and can reroute method calls to it from attribute-interceptor methods to invoke higher default behavior. Per Chapter 30:

object.__setattr__(self, attr, value)

We’ll code examples of such rerouting later in the book; for now, let’s move on to something a bit more tangible and our next topic in this OOP jamboree.

Slots: Attribute Declarations

First off, we’ve noted the implications of slots several times in this part of the book. In short, by assigning an iterable of attribute name strings to a special __slots__ class attribute, we can enable a class to both limit the set of legal attributes that instances of the class will have and optimize memory usage and possibly program speed. As you’ll find, though, slots should be used only in applications that clearly warrant the added complexity. They will complicate your code, may complicate or break code you may use, and rigidly require universal deployment to be effective.

>>> class Limiter(object):
        __slots__ = ['age', 'name', 'job']                # Slots "declaration"

>>> I = Limiter()
>>> I.age                                                 # Must assign before use
AttributeError: 'Limiter' object has no attribute 'age' 

>>> I.age = 40                                            # Looks like instance data
>>> I.age
40
>>> I.ape = 1000                                          # Fails: not in __slots__
AttributeError: 'Limiter' object has no attribute 'ape'

>>> class C:
        __slots__ = ['a', 'b']           # __slots__ means no __dict__ by default

>>> I = C()
>>> I.a = 1
>>> I.a
1
>>> I.__dict__
AttributeError: 'C' object has no attribute '__dict__'. Did you mean: '__dir__'?

>>> getattr(I, 'a')
1
>>> setattr(I, 'b', 2)                   # But getattr() and setattr() still work
>>> I.b
2
>>> 'a' in dir(I)                        # And dir() finds slot attributes too
True
>>> 'b' in dir(I)                        # Though __dict__ access will fail
True
>>> I.__dict__
AttributeError: 'C' object has no attribute '__dict__'. Did you mean: '__dir__'?

>>> class D:                             
        __slots__ = ['a', 'b']
        def __init__(self):
            self.d = 4                   # Cannot add new names if no __dict__

>>> I = D()
AttributeError: 'D' object has no attribute 'd'

>>> class D:
        __slots__ = ['a', 'b', '__dict__']    # Name __dict__ to include one too
        c = 3                                 # Class attrs work normally
        def __init__(self):
            self.d = 4                        # d stored in __dict__, a is a slot

>>> I = D()
>>> I.d
4
>>> I.c
3
>>> I.a                          # All instance attrs undefined until assigned
AttributeError: 'D' object has no attribute 'a' 
>>> I.a = 1
>>> I.b = 2

>>> I.__dict__                   # Some objects have both __dict__ and slot names
{'d': 4}                         # getattr() can fetch either type of attr
>>> I.__slots__
['a', 'b', '__dict__']
>>> getattr(I, 'a'), getattr(I, 'c'), getattr(I, 'd')    # Fetches all 3 forms
(1, 3, 4)

>>> for attr in list(I.__dict__) + I.__slots__:          # Wrong...
        print(attr, '=>', getattr(I, attr))

>>> for attr in list(getattr(I, '__dict__', [])) + getattr(I, '__slots__', []):
        print(attr, '=>', getattr(I, attr))

d => 4
a => 1                                                   # Less wrong...
b => 2
__dict__ => {'d': 4}

>>> class E:
        __slots__ = ['c', 'd']            # Superclass has slots

>>> class D(E):
        __slots__ = ['a', '__dict__']     # But so does its subclass

>>> I = D()                               # The instance gets the union of each
>>> dir(I)
[…names omitted…, 'a', 'c', 'd']
>>> I.a = 1; I.b = 2; I.c = 3             # slots: a, c, __dict__: b
>>> I.a, I.c
(1, 3)

>>> E.__slots__                           # But __slots__ not concatenated
['c', 'd']
>>> D.__slots__
['a', '__dict__']
>>> I.__slots__                           # Instance inherits *lowest* __slots__
['a', '__dict__']
>>> I.__dict__                            # And has its own an attr dict
{'b': 2}

>>> for attr in list(getattr(I, '__dict__', [])) + getattr(I, '__slots__', []):
        print(attr, '=>', getattr(I, attr))

b => 2                                    # Other superclass slots missed!
a => 1
__dict__ => {'b': 2}

>>> dir(I)                                # But dir() includes all slot names
[…names omitted…, 'a', 'b', 'c', 'd']

>>> class Slotful:
        __slots__ = ['a', 'b', '__dict__']
        def __init__(self, data):
            self.c = data

>>> I = Slotful(3)
>>> I.a, I.b = 1, 2
>>> I.a, I.b, I.c                            # Normal attribute fetch
(1, 2, 3)

>>> I.__dict__                               # Both __dict__ and slots storage
{'c': 3}
>>> [x for x in dir(I) if not x.startswith('__')]
['a', 'b', 'c']

>>> I.__dict__['c']                          # __dict__ is only one attr source
3
>>> getattr(I, 'c'), getattr(I, 'a')         # dir+getattr is broader than __dict__
(3, 1)                                       # applies to slots, properties, descrip

>>> for a in (x for x in dir(I) if not x.startswith('__')):
        print(a, '=>', getattr(I, a))

a 1
b 2
c 3

>>> class C: pass                        # Bullet 1: slots in sub but not super
>>> class D(C): __slots__ = ['a']        # Makes instance dict for nonslots
>>> I = D()                              # But slot name still managed in class
>>> I.a = 1; I.b = 2
>>> I.__dict__
{'b': 2}
>>> D.__dict__.keys()
dict_keys([… '__slots__', 'a', …])

>>> class C: __slots__ = ['a']           # Bullet 2: slots in super but not sub
>>> class D(C): pass                     # Makes instance dict for nonslots
>>> I = D()                              # But slot name still managed in class
>>> I.a = 1; I.b = 2
>>> I.__dict__
{'b': 2}
>>> C.__dict__.keys()
dict_keys([… '__slots__', 'a', …])

>>> class C: __slots__ = ['a']           # Bullet 3: only lowest slot accessible
>>> class D(C): __slots__ = ['a']        # Superclass slot 'a' is pointless

>>> class C: __slots__ = ['a']; a = 99   # Bullet 4: no class-level defaults
ValueError: 'a' in __slots__ conflicts with class variable
>>> class C: __slots__ = ['a']           # Bullet 5: only one nonempty in mixins
>>> class D: __slots__ = ['a']           # Use empty slots or omit in all but one
>>> class E(C, D): pass
TypeError: multiple bases have instance lay-out conflict

>>> class C: __slots__ = ['a']           # Assumes universal use, differing names
>>> class D(C): __slots__ = ['b']
>>> I = D()                              # And may break code and tools you use
>>> I.a = 1; I.b = 2
>>> I.__dict__
AttributeError: 'D' object has no attribute '__dict__'. Did you mean: '__dir__'?
>>> C.__dict__.keys(), D.__dict__.keys()
(dict_keys([… '__slots__', 'a', …]), dict_keys([… '__slots__', 'b', …]))

class C(ListTree): pass
I = C()                                        # OK: no __slots__ used
print(I)

class C(ListTree): __slots__ = ['a', 'b']      # OK: superclass produces __dict__
I = C()
I.c = 3
print(I)                                       # Displays c at I, a and b at C

class A: __slots__ = ['a']                     # Both OK by bullet 1 above
class B(A, ListTree): pass
print(B())

class A: __slots__ = ['a']
class B(A, ListTree): __slots__ = ['b']        # Displays b at B, a at A
print(B())

import timeit
base = """
Is = []
for i in range(1000):
    I = C()
    I.a = 1; I.b = 2; I.c = 3; I.d = 4
    t = I.a + I.b + I.c + I.d
    Is.append(I)
"""

stmt = """
class C:
    __slots__ = ['a', 'b', 'c', 'd']
""" + base
print('Slots   =>', end=' ')
print(min(timeit.repeat(stmt, number=1000, repeat=5)))

stmt = """
class C:
    pass
""" + base
print('Nonslots=>', end=' ')
print(min(timeit.repeat(stmt, number=1000, repeat=5)))

$ python3 slots-test.py
Slots   => 0.17895982996560633
Nonslots=> 0.18887511501088738

Properties: Attribute Accessors

Our next attribute-related topic is properties—a mechanism that provides another way for classes to define methods called automatically for access or assignment to instance attributes. This feature is similar to “getters” and “setters” in languages like Java and C#, but in Python is generally best used sparingly as a way to add accessors to attributes after the fact as needs evolve and warrant. Where needed, though, properties allow attribute values to be computed dynamically without requiring method calls at the point of access.

Though properties cannot support generic attribute routing goals, at least for specific attributes, they are an alternative to some traditional uses of the __getattr__ and __setattr__ overloading methods we first studied in Chapter 30. Properties can have a similar effect to these two methods but, by contrast, incur an extra method call only for accesses to names that require dynamic computation—other nonproperty names are accessed normally with no extra calls. Although __getattr__ is invoked only for undefined names, the __setattr__ method is instead called for assignment to every attribute.

Properties and slots are related, too, but serve different goals. Both implement instance attributes that are not physically stored in instance namespace dictionaries—a sort of “virtual” attribute—and both are based on the notion of class-level attribute descriptors. In contrast, slots manage instance storage, while properties intercept access and compute values arbitrarily. Because their underlying descriptor implementation tool is too advanced for us to cover here, properties and descriptors both get full treatment in the online-only “Managed Attributes” chapter.

>>> class WithOperators:
        def __getattr__(self, name):   # On undefined attr
            if name == 'age':
                return 40
            else:
                raise AttributeError(name)

>>> x = WithOperators()
>>> x.age                         # Runs __getattr__
40
>>> x.name                        # Runs __getattr__
AttributeError: name

>>> class WithProperties:
        def getage(self):
            return 40
        age = property(getage)    # (get?, set?, del?, docs?), or @

>>> x = WithProperties()
>>> x.age                         # Runs getage
40
>>> x.name                        # Normal fetch
AttributeError: 'WithProperties' object has no attribute 'name'

>>> class WithProperties:    
        def getage(self):
            print('get age')            
            return 40
        def setage(self, value):
            print('set age:', value)
            self._age = value
        age = property(getage, setage)

>>> x = WithProperties()
>>> x.age                         # Runs getage
get age
40
>>> x.age = 42                    # Runs setage
set age: 42
>>> x._age                        # Normal fetch:  no getage call
42
>>> x.job = 'hacker'              # Normal assign: no setage call
>>> x.job                         # Normal fetch:  no getage call
'hacker'

>>> class WithOperators:
        def __getattr__(self, name):            # On undefined attr
            if name == 'age':
                print('get age')
                return 40
            else:
                raise AttributeError(name)
        def __setattr__(self, name, value):     # On all assignments
            print('set:', name, value)
            if name == 'age':
                self.__dict__['_age'] = value   # Or object.__setattr__(self, ...)
            else:
                self.__dict__[name] = value

>>> x = WithOperators()
>>> x.age                         # Runs __getattr__
get age
40
>>> x.age = 41                    # Runs __setattr__
set: age 41
>>> x._age                        # Defined: no __getattr__ call
41
>>> x.job = 'coder'               # Runs __setattr__ again
set: job coder
>>> x.job                         # Defined: no __getattr__ call
'coder'

class WithProperties:
    @property                     # Coding properties with decorators: ahead
    def age(self):                # On instance.age
        …
    @age.setter
    def age(self, value):         # On instance.age = value
        …

__getattribute__ and Descriptors: Attribute Tools

To complete our attribute-tools collection, the __getattribute__ operator-overloading method intercepts all attribute references, not just undefined references. This makes it more potent than its __getattr__ cousin we used in the prior section, but also trickier to use—it’s prone to loops much like __setattr__, but in different ways.

For more specialized attribute interception goals, in addition to properties and operator-overloading methods, Python provides attribute descriptors—classes with __get__ and __set__ methods, assigned to class attributes and inherited by instances, that intercept read and write accesses to specific attributes. As a preview, here’s one of the simplest descriptors you’re likely to encounter:

>>> class AgeDesc:
        def __get__(self, instance, owner): return 40
        def __set__(self, instance, value): instance._age = value

>>> class WithDescriptors:
        age = AgeDesc()           # Assign descriptor instance

>>> x = WithDescriptors()
>>> x.age                         # Runs AgeDesc.__get__
40
>>> x.age = 42                    # Runs AgeDesc.__set__
>>> x._age                        # Normal fetch: no AgeDesc call
42

Descriptors have access to state-information attributes in instances of themselves as well as their client class and are, in a sense, a more general form of properties. In fact, properties are a simplified way to define a specific type of descriptor—one that runs functions on access. Descriptors are also used to implement the slots feature we met earlier, among other Python tools, and are afforded special cases in attribute inheritance alluded to earlier in this chapter.

Because __getattribute__ and descriptors are too substantial to present here, we’ll defer the rest of their coverage, as well as much more on properties, to the online-only “Managed Attributes” chapter. We’ll also employ them in examples in the online-only “Decorators” chapter and study how they factor into inheritance in Chapter 40. Here, the topics tour is moving on.

Why the Special Methods?

As we’ve seen, a class’s method is normally passed an instance object in its first argument to serve as the implied subject of the method call—that’s the “object” in “object-oriented programming.” Though much less common, there are two formal ways to temper this model. Before we get to the syntax, let’s clarify why this might matter to you.

Sometimes, programs need to process data associated with classes instead of instances. Consider keeping track of the number of instances created from a class or maintaining a list of all of a class’s instances that are currently in use. This type of information and its processing are associated with the class rather than its instances. That is, the information is usually stored on the class itself and processed apart from any instance.

For such tasks, simple functions coded outside a class might suffice—because they can access class attributes through the class name, they have access to class data, and never require access to an instance. However, to better associate such code with a class and to allow such processing to be customized with inheritance as usual, it would be better to code these types of functions inside the class itself. To make this work, we need methods in a class that are not passed, and do not expect, a self instance argument.

Per the prior chapter, methods accessed through the class are plain functions that meet some of this need but fail if accessed through an instance: the resulting bound method passes an instance in calls, even if the plain function doesn’t expect one. To address, Python provides static methods—plain functions that are nested in a class and never expect nor receive an automatic self argument, regardless of how they are called. They’re optional for methods only ever accessed through classes but needed for access through instances.

Although less commonly used, Python also supports class methods—methods of a class that are passed a class object in their first argument instead of an instance, regardless of whether they are called through an instance or a class. Such methods can access class data through their class argument—what we’ve called self thus far—even if called through an instance. Normal methods, sometimes called instance methods, still receive a subject instance when called; static and class methods do not.

Plain-Function Methods

To demo the preceding ideas, let’s suppose that we want to use class attributes to count how many instances are generated from a class. Example 32-5, hack1.py, makes a first attempt—its class has a counter stored as a class attribute, a constructor that bumps up the counter by one each time a new instance is created, and a method that displays the counter’s value. Remember, class attributes are stored just once on a class and shared by all instances; storing the counter this way ensures that it effectively spans all instances.

class Hack:
    numInstances = 0
    def __init__(self):
        Hack.numInstances += 1
    def printNumInstances():
        print('Number of instances created:', Hack.numInstances)

The printNumInstances method is designed to process class data, not instance data—it’s about all the instances, not any one in particular. Because of that, we want to be able to call it without having to pass an instance. Indeed, we don’t want to make an instance to fetch the number of instances because this would change the number of instances we’re trying to fetch! In other words, we want a self-less “static” method.

Whether this code’s printNumInstances works or not, though, depends on which way you call the method—through the class or through an instance. Calls to self-less methods made through classes work because they produce plain functions, but calls from instances produce bound methods and fail:

$ python3
>>> from hack1 import Hack
>>> a, b, c = Hack(), Hack(), Hack()     # Make three instances

>>> Hack.printNumInstances()             # Okay to call from instance – only!
Number of instances created: 3
>>> a.printNumInstances()
TypeError: Hack.printNumInstances() takes 0 positional arguments but 1 was given

Calls to instance-less methods like printNumInstances made through the class work, but calls made through an instance fail because an instance is automatically passed to a method that does not have an argument to receive it. If you’re able to stick with calling self-less methods through classes only, you already have a static method. However, to allow self-less methods to be called through instances, you need to either adopt other designs or mark such methods as special. Let’s look at both options in turn.

Static Method Alternatives

Short of marking a self-less method as special, you can sometimes achieve similar results with different coding structures. For example, if you just want to call functions that access class members without an instance, perhaps the simplest idea is to use normal functions outside the class, not class methods. This way, an instance isn’t expected in the call. The mutation in Example 32-6 illustrates.

def printNumInstances():
    print('Number of instances created:', Hack.numInstances)

class Hack:
    numInstances = 0
    def __init__(self):
        Hack.numInstances += 1

Because the class name is accessible to the simple function as a global variable, this works fine. Also, note that the name of the function becomes global, but only to this single module; it will not clash with names in other files:

>>> import hack2 as hack
>>> a = hack.Hack()
>>> b = hack.Hack()
>>> c = hack.Hack()
>>> hack.printNumInstances()           # But function may be too far removed
Number of instances created: 3         # And cannot be changed via inheritance
>>> hack.Hack.numInstances
3

Prior to static methods in Python, this structure was the general prescription. Because Python already provides modules as a namespace-partitioning tool, one could argue that there’s not typically any need to package functions in classes unless they implement object behavior. Simple functions within modules like the one here do much of what instance-less class methods could and are already associated with the class because they live in the same module.

This approach, though, may be subpar. For one thing, it adds to this file’s scope an extra name that is used only for processing a single class. For another, the function is not directly associated with the class by structure; in fact, its def could be hundreds of lines away. Worse, simple functions like this cannot be customized by inheritance since they live outside a class’s namespace: subclasses cannot directly replace or extend such a function by redefining it.

We might also try to make this example work by simply using a normal method and always calling it through an instance, as usual. Unfortunately, such an approach is completely unworkable if we don’t have an instance available, and making an instance changes the class data, as noted earlier. A better solution would be to somehow mark a method inside a class as never requiring an instance. The next section shows how.

Using Static and Class Methods

To designate a self-less method that may be called through either the class or its instances, classes can simply call the built-in functions staticmethod and classmethod. Both mark a function object as special—requiring no instance for the former and requiring a class argument for the latter. Example 32-7 shows how.

class Methods:
    def imeth(self, x):            # Instance method: passed a self
        print([self, x])           # Always expects a self instance

    def smeth(x):                  # Static method: no instance passed
        print([x])                 # Also a plain function from the class

    def cmeth(cls, x):             # Class method: gets class, not instance
        print([cls, x])            # Always expects a class, not instance

    smeth = staticmethod(smeth)    # Make smeth a static method (or use @: ahead)
    cmeth = classmethod(cmeth)     # Make cmeth a class method (or use @: ahead)

Notice how the last two assignments in this code simply reassign (a.k.a. rebind) the method names smeth and cmeth. Attributes are created and changed by any assignment in a class statement, so these final assignments simply overwrite the assignments made earlier by the defs. As you’ll see in a few moments, the special @ decorator syntax works here as an alternative to this just as it does for properties—but makes little sense unless you first understand the assignment form here that it automates.

Technically, Python supports three kinds of class-related methods with differing argument protocols:

• Instance methods, passed a self instance object (the default)

• Static methods, passed no extra instance object (via staticmethod)

• Class methods, passed a class object (via classmethod, and inherent in metaclasses)

Moreover, simple functions in a class also serve the role of static methods without requiring any extra protocol when called through a class object only. The allmethods.py module illustrates all three method types, so let’s expand on these in turn.

Instance methods are the normal and default case that we’ve used in this book so far. An instance method must always be called with an instance object. When you call it through an instance, Python passes the instance to the first (leftmost) argument automatically; when you call it through a class, you must pass along the instance manually:

>>> from allmethods import Methods     # Normal instance methods
>>> obj = Methods()                    # Callable through instance or class
>>> obj.imeth(1)                       # Becomes imeth(obj, 1)
[<allmethods.Methods object at 0x1015a79b0>, 1] 
>>> Methods.imeth(obj, 2)
[<allmethods.Methods object at 0x1015a79b0>, 2]

Static methods, by contrast, are called without an instance argument. Unlike simple functions outside a class, their names are local to the scopes of the classes in which they are defined, and they may be looked up by inheritance. Instance-less functions can be called through a class normally, but using the staticmethod built-in allows such methods to also be called through an instance. That is, the first of the following works without the staticmethod in the class but the second does not:

>>> Methods.smeth(3)                   # Static method: call through class
[3]                                    # No instance passed or expected
>>> obj.smeth(4)                       # Static method: call through instance
[4]                                    # Instance not passed – requires staticmethod

Class methods are similar, but Python automatically passes the class (not an instance) to a class method’s first (leftmost) argument, whether it is called through a class or an instance:

>>> Methods.cmeth(5)                   # Class method: call through class
[<class 'allmethods.Methods'>, 5]      # Becomes cmeth(Methods, 5)
>>> obj.cmeth(6)                       # Class method: call through instance
[<class 'allmethods.Methods'>, 6]      # Becomes cmeth(Methods, 6)

In Chapter 40, you’ll also find that metaclass methods—an advanced and technically distinct method type used in the secondary class trees of types—behave similarly to the explicitly declared class methods we’re exploring here.

Counting Instances with Static Methods

Now, given these built-ins, Example 32-8 codes the static method equivalent of this section’s instance-counting example—it marks the method as special, so it will never be passed an instance automatically.

class Hack:
    numInstances = 0                         # Use static method for class data
    def __init__(self):
        Hack.numInstances += 1
    def printNumInstances():
        print('Number of instances:', Hack.numInstances)
    printNumInstances = staticmethod(printNumInstances)

Using the static method built-in, our code now allows the self-less method to be called through the class or any instance of it:

>>> from hack_static import Hack
>>> a, b, c = Hack(), Hack(), Hack()
>>> Hack.printNumInstances()                 # Call as simple function
Number of instances: 3
>>> a.printNumInstances()                    # Instance argument not passed
Number of instances: 3

Compared to simply moving printNumInstances outside the class, as prescribed earlier, this version requires an extra staticmethod call (or an @ line you’ll meet ahead). However, it also localizes the function name in the class scope (so it won’t clash with other names in the module); moves the function code closer to where it is used (inside the class statement); and allows subclasses to customize the static method with inheritance—a more convenient and powerful approach than importing functions from the files in which superclasses are coded. The following subclass illustrates (this continues the prior session, so the count is already 3 at the start):

>>> class Sub(Hack):
        def printNumInstances():             # Override a static method
            print('Extra stuff...')          # But call back to original
            Hack.printNumInstances()
        printNumInstances = staticmethod(printNumInstances)
 
>>> a, b = Sub(), Sub()
>>> a.printNumInstances()                    # Call from subclass instance
Extra stuff...
Number of instances: 5
>>> Sub.printNumInstances()                  # Call from subclass itself
Extra stuff...
Number of instances: 5
>>> Hack.printNumInstances()                 # Call original version
Number of instances: 5

Moreover, classes can inherit the static method without redefining it—it is run without an instance, regardless of where it is defined in a class tree:

>>> class Other(Hack): pass                  # Inherit static method verbatim

>>> c = Other()
>>> c.printNumInstances()
Number of instances: 6

Notice how this also bumps up the superclass’s instance counter because its constructor is inherited and run—a behavior that begins to encroach on the next section’s subject.

Counting Instances with Class Methods

Interestingly, a class method can do similar work here—Example 32-9 has the same behavior as the static method version listed earlier, but it uses a class method that receives the instance’s class in its first argument. Rather than hardcoding the class name, the class method uses the automatically passed class object generically.

class Hack:
    numInstances = 0                         # Use class method instead of static
    def __init__(self):
        Hack.numInstances += 1
    def printNumInstances(cls):
        print('Number of instances:', cls.numInstances)
    printNumInstances = classmethod(printNumInstances)

This class is used in the same way as the prior versions, but its printNumInstances method receives the Hack class, not the instance, when called from either the class or an instance:

>>> from hack_class import Hack
>>> a, b = Hack(), Hack()
>>> a.printNumInstances()                    # Passes class to first argument
Number of instances: 2
>>> Hack.printNumInstances()                 # Also passes class to first argument
Number of instances: 2

When using class methods, though, keep in mind that they receive the most specific (i.e., lowest) class of the call’s subject. This has some subtle implications when trying to update class data through the passed-in class. To demo, Example 32-10 subclasses to customize the same way we did for static methods in the prior section, and augments Hack.printNumInstances to also trace its cls argument.

class Hack:
    numInstances = 0                         # Trace class passed in
    def __init__(self):
        Hack.numInstances += 1
    def printNumInstances(cls):
        print('Number of instances:', cls.numInstances, cls)
    printNumInstances = classmethod(printNumInstances)

class Sub(Hack):
    def printNumInstances(cls):              # Override a class method
        print('Extra stuff...', cls)         # But call back to original
        Hack.printNumInstances()
    printNumInstances = classmethod(printNumInstances)

class Other(Hack): pass                      # Inherit class method verbatim

Running this in a REPL reveals that the lowest class is passed in whenever a class method is run—even for subclasses that have no class methods of their own:

>>> from hack_class2 import Hack, Sub, Other
>>> x = Sub()
>>> y = Hack()

>>> x.printNumInstances()                           # Call from subclass instance
Extra stuff... <class 'hack_class2.Sub'>
Number of instances: 2 <class 'hack_class2.Hack'>

>>> Sub.printNumInstances()                         # Call from subclass itself
Extra stuff... <class 'hack_class2.Sub'>
Number of instances: 2 <class 'hack_class2.Hack'>

>>> y.printNumInstances()                           # Call from superclass instance
Number of instances: 2 <class 'hack_class2.Hack'>

In the first call here, a class method call is made through an instance of the Sub subclass, and Python passes the lowest class, Sub, to the class method. All is well in this case—since Sub’s redefinition of the method calls the Hack superclass’s version explicitly, the superclass method in Hack receives its own class in its first argument. But watch what happens for an object that inherits the class method verbatim:

>>> z = Other()                                     # Call from lower sub's instance
>>> z.printNumInstances()
Number of instances: 3 <class 'hack_class2.Other'>

This last call here passes Other to Hack’s class method. This works in this example because fetching the counter finds it in Hack by class inheritance. If this method tried to assign to the passed class’s data, though, it would update Other, not Hack! In this specific case, Hack is probably better off hardcoding its own class name to update its data if it means to count instances of all its subclasses, too, rather than relying on the passed-in class argument.

class Hack:
    numInstances = 0
    def count(cls):                    # Per-class instance counters
        cls.numInstances += 1          # cls is lowest class above instance
    def __init__(self):
        self.count()                   # Passes self.__class__ to count
    count = classmethod(count)

class Sub(Hack):
    numInstances = 0
    def __init__(self):                # Redefines __init__ (to demo)
        Hack.__init__(self)

class Other(Hack):                     # Inherits __init__
    numInstances = 0

>>> from hack_class3 import Hack, Sub, Other
>>> x = Hack()
>>> y1, y2 = Sub(), Sub()
>>> z1, z2, z3 = Other(), Other(), Other()
>>> x.numInstances, y1.numInstances, z1.numInstances             # Per-class data!
(1, 2, 3)
>>> Hack.numInstances, Sub.numInstances, Other.numInstances
(1, 2, 3)

Function Decorator Basics

Syntactically, a function decorator is a sort of runtime declaration about the function that follows it. A function decorator is coded on a line by itself just before the def statement that defines a function or method. It consists of the @ symbol, followed by a metafunction—a plain function (or other callable object) of one argument, which is passed and manages another function. The code following the @ is usually the name of a metafunction with an optional arguments list, but as of Python 3.9, it can be any expression returning a one-argument function. Listing multiple decorators on consecutive lines allows them to nest, as you’ll see later in this book.

For example, the prior section’s methods may be coded with decorator syntax like this:

class C:
   @staticmethod                    # Function decoration syntax
   def meth():
       …

Internally, this syntax has the same effect as the following—passing the function through the decorator and assigning the result back to the original name:

class C:
   def meth():
       …
   meth = staticmethod(meth)        # Name rebinding equivalent

Decoration rebinds the method name to the decorator’s result. The net effect is that calling the method function’s name later actually triggers the result of its staticmethod decorator first. Because a decorator can return any sort of object, this allows the decorator to insert a layer of logic to be run on every later call. The decorator function is free to return either the original function itself or a new proxy object that saves the original function passed to the decorator to be invoked indirectly after the extra logic layer runs.

With this addition, Example 32-12 is a better way to code our static method code of Example 32-8.

class Hack:
    numInstances = 0
    def __init__(self):
        Hack.numInstances += 1

    @staticmethod
    def printNumInstances():
        print('Number of instances:', Hack.numInstances)

Here is this example in action works as before:

>>> from hack_static_deco import Hack
>>> a, b, c = Hack(), Hack(), Hack()
>>> Hack.printNumInstances()              # Calls from classes and instances work
Number of instances: 3
>>> a.printNumInstances()
Number of instances: 3

Because they also accept and return functions, the classmethod and property built-in functions may be used as decorators in the same way—as in Example 32-13, which demos all three built-in decorators previewed earlier.

class Methods:
    def imeth(self, x):            # Normal instance method: passed a self
        print([self, x])

    @staticmethod
    def smeth(x):                  # Static: no instance passed
        print([x])

    @classmethod
    def cmeth(cls, x):             # Class: gets class, not instance
        print([cls, x])

    @property                      # Property: computed on fetch
    def name(self):
        return 'Pat ' + self.__class__.__name__

Running this live in a REPL proves the point:

>>> from alldecorators import Methods
>>> obj = Methods()
>>> obj.imeth(1)
[<alldecorators.Methods object at 0x10d2839b0>, 1]
>>> obj.smeth(2)
[2]
>>> obj.cmeth(3)
[<class 'alldecorators.Methods'>, 3]
>>> obj.name
'Pat Methods'

Bear in mind that staticmethod and its kin here are still built-in functions; they may be used in decoration syntax just because they take a function as an argument and return a callable to which the original function name can be rebound. In fact, any such function can be used in this way—even user-defined functions we code ourselves, as the next section explains.

A First Look at User-Defined Function Decorators

Although Python provides a handful of built-in functions that can be used as decorators, we can also write custom decorators of our own. Because of their wide utility, we’re going to devote an entire chapter to coding decorators in the final part of this book. As a quick example, though, let’s look at a simple user-defined decorator at work.

Recall from Chapter 30 that the __call__ operator-overloading method implements a function-call interface for class instances. Example 32-14 uses this to code a call proxy class that saves the decorated function in the instance and catches calls to the original name. Because this is a class, it also has state information—a counter of calls made.

class tracer:
    def __init__(self, func):          # Remember original, init counter
        self.calls = 0
        self.func  = func
    def __call__(self, *args):         # On later calls: add logic, run original
        self.calls += 1
        print(f'call {self.calls} to {self.func.__name__}')
        return self.func(*args)

@tracer                                # Same as hack = tracer(hack)
def hack(a, b, c):                     # Wrap hack in a decorator object
    return a + b + c

if __name__ == '__main__':
    print(hack(1, 2, 3))               # Really calls the tracer wrapper object
    print(hack('a', 'b', 'c'))         # Invokes __call__ in class

Because the hack function is run through the tracer decorator, when the original hack name is called, it actually triggers the __call__ method in the class. This method counts and logs the call and then dispatches it to the original wrapped function. Note how the *name argument syntax is used to pack and unpack the passed-in arguments; because of this, this decorator can be used to wrap any function with any number of positional arguments.

The net effect, again, is to add a layer of logic to the original hack function. When run, the first output line comes from the tracer class, and the second gives the return value of the hack function itself:

$ python3 tracer1.py
call 1 to hack
6
call 2 to hack
abc

Trace through this example’s code for more insight. As it is, this decorator works for any function that takes positional arguments, but it does not handle keyword arguments and cannot decorate class-level method functions (in short, for methods, its __call__ would be passed a tracer instance only). As you’ll learn in Part VIII, there are a variety of ways to code function decorators, including nested def statements, and some of the alternatives are better suited to methods than the version shown here.

For example, by using nested functions with enclosing scopes for state instead of callable class instances with attributes, function decorators often become more broadly applicable to class-level methods too. We’ll postpone the full details on this, but Example 32-15 provides a brief look at this closure-based coding model; it uses function attributes for counter state for portability but could also leverage variables and nonlocal instead.

def tracer(func):                      # Remember original
    def oncall(*args):                 # On later calls
        oncall.calls += 1
        print(f'call {oncall.calls} to {func.__name__}')
        return func(*args)
    oncall.calls = 0
    return oncall

class C:
    @tracer
    def hack(self, a, b, c): return a + b + c

if __name__ == '__main__':
    x = C()
    print(x.hack(1, 2, 3))
    print(x.hack('a', 'b', 'c'))

The example’s output is the same as its predecessor but reflects a decorated class method; more on this later.

A First Look at Class Decorators and Metaclasses

Python later generalized decorators, allowing them to be applied to classes as well as functions. In short, class decorators are similar to function decorators, but they are run at the end of a class statement to rebind a class name to a callable. As such, they can be used to either manage classes just after they are created or insert a layer of wrapper logic to manage instances when they are later created. Symbolically, the code structure:

def decorator(aClass): …

@decorator                       # Class decoration syntax
class C: …

is mapped to the following equivalent:

def decorator(aClass): …

class C: …                       # Name rebinding equivalent
C = decorator(C)

The class decorator is free to augment the class itself or return a proxy object that intercepts later instance construction calls. For example, in the code of the section “Counting instances per class with class methods”, we could use this hook to automatically augment the classes with instance counters and any other data required:

def count(aClass):
    aClass.numInstances = 0
    return aClass                # Return class itself instead of a wrapper

@count
class Hack: …                    # Same as Hack = count(Hack)

@count
class Sub(Hack): …               # numInstances = 0 not needed here

In fact, as coded, this decorator can be applied to classes or functions—it happily returns the object being defined in either context after initializing the object’s attribute:

@count
def hack(): pass                 # Like hack = count(hack)

@count
class Hack: pass                 # Like Hack = count(Hack)

hack.numInstances                # Both are set to zero
Hack.numInstances

Though this decorator manages a function or class itself, as detailed later in this book, class decorators can also manage an object’s entire interface by intercepting construction calls and wrapping the new instance object in a proxy that deploys attribute accessor tools to intercept later requests—a multilevel coding technique we’ll use to implement class attribute privacy in the online-only “Decorators” chapter. Here’s a preview of the model:

def decorator(cls):                             # On @ decoration
    class Proxy:
        def __init__(self, *args):              # On instance creation: make a cls
            self.wrapped = cls(*args)
        def __getattr__(self, name):            # On attribute fetch: extra ops here
            return getattr(self.wrapped, name)
    return Proxy

@decorator
class C: …          # Like C = decorator(C)
X = C()             # Makes a Proxy that wraps a C, and catches later X.attr

Finally, metaclasses, mentioned briefly earlier in this chapter, are a similarly advanced class-based tool whose roles often intersect with those of class decorators. They provide an alternate model, which routes the creation of a class object to a subclass of the top-level type class (normally), at the conclusion of a class statement:

class Meta(type):
    def __new__(meta, classname, supers, classdict):
        …extra logic + class creation via type call…

class C(metaclass=Meta):
    …my creation routed to Meta…            # Like C = Meta('C', (), {…})

Python calls a class’s metaclass to create the new class object, passing in the data defined during the class statement’s run; if omitted, the metaclass simply defaults to the type class we explored earlier. Abstractly speaking, here’s what happens at the end of class statements having explicit metaclasses like the preceding:

 classname = Meta(classname, superclasses, attributedict)

To manage the creation or initialization of a new class object, a metaclass generally redefines the __new__ or __init__ method of the type class that intercepts this call by default. The net effect, as with class decorators, is to define code to be run automatically at class creation time. Here, this binds the class name to the result of a call to a user-defined metaclass. In fact, a metaclass need not be a class at all—a possibility we’ll explore later that blurs some of the distinction between this tool and decorators and even qualifies the two as functionally equivalent in some roles.

Both schemes, class decorators and metaclasses, are free to augment a class or return an arbitrary object to replace it—a hook with almost limitless class-based customization possibilities. As you’ll learn later, metaclasses may also define methods that process their instance classes rather than normal instances of them—a technique that’s similar (if not redundant) to class methods and might be emulated by methods and data in class decorator proxies, or even a class decorator that returns a metaclass instance.

Such mind-bending concepts, however, require Chapter 40’s conceptual groundwork (and quite possibly sedation).

For More Details

Naturally, there’s more to the decorator and metaclass stories than shown here. Although they are a general mechanism whose usage may be required by some packages, coding new user-defined decorators and metaclasses is an advanced topic of interest primarily to tool writers, not application programmers. Because of this, this book defers additional coverage until its final and optional part:

• The online-only “Managed Attributes” chapter shows how to code properties using function decorator syntax in more depth.

• The online-only “Decorators” chapter focuses on decorators, and includes more comprehensive examples.

• Chapter 40 covers metaclasses, and more on the class and instance management story.

Although these chapters cover advanced topics, they’ll also provide us with a chance to see Python at work in substantial examples. For now, let’s move on to our final class-related topic.

The super Basics

First off, let’s review the explicit alternative that’s more closely in step with Python’s own idioms. As we’ve seen in this book so far, it’s always possible to reference a superclass’s attributes—whether method or data—by naming their desired source class explicitly:

>>> class C:
        def act(self):
            print('hack')

>>> class D(C):
        def act(self):
            C.act(self)         # Name superclass explicitly, pass self
            print('code')

>>> I = D()
>>> I.act()
hack
code

In single-inheritance trees like this one, the super alternative seems relatively straightforward at first glance: its most common form in the following automatically selects the calling class’s superclass generically and implicitly when an attribute is later fetched. An explicit call like class.method(self), for example, becomes an implicit super().method(), as in our example:

>>> class C: 
        def act(self):
            print('hack')

>>> class D(C):
        def act(self):
            super().act()       # Reference superclass implicitly, omit self
            print('code')

>>> I = D()
>>> I.act()
hack
code

This works as advertised and may minimize work—you don’t need to list self in the call, don’t need to update the call if D’s superclass changes in the future, and don’t need to code long superclass names or package-import paths.

The super Details

If you study the preceding code closely, though, you’ll realize that there’s something odd going on here. The super call somehow knows about the class, its superclass, and the self instance, even though none are present in its call. The backstory involves MROs, a proxy, and an algorithm that are required reading for super aspirants of all kinds.

super()
super(class-containing-the-super-call, method-self-argument)

class D(C):
    def act(self):
        super(D, self).act()    # Works the same as super().act()
        print('code')           # And D is available as __class__

>>> super                       # A "magic" proxy object that routes later calls
<class 'super'>
>>> super()                     # This form has no meaning outside a method
RuntimeError: super(): no arguments

>>> class E(C):
        def method(self):
            proxy = super()     # self is implicit in super - only!
            return proxy
 
>>> prx = E().method()          # The normally hidden proxy object
>>> prx
<super: <class 'E'>, <E object>>

>>> prx.act()    # Find act on MRO past hidden E, bind with hidden self, call (!)
Hack

>>> E.__mro__
(<class '__main__.E'>, <class '__main__.C'>, <class 'object'>)
>>> E().__class__.__bases__[0]
<class '__main__.C'>

>>> class A:
        def act(self): print('A')

>>> class B:
        def act(self): print('B')

>>> class C(B):
        def act(self):
            super().act()         # super applied to a single-inheritance tree

>>> C().act()                     # Make an instance and call its method
B

>>> class C(B, A):                # Add an A mix-in class with the same method
        def act(self):
            super().act()         # Doesn't fail on conflicts - picks just one
>>> C().act()
B

>>> class C(A, B):
        def act(self):
            super().act()         # If A is listed first, B.act() is no longer run
>>> C().act()
A

>>> class C(A, B):                # Explicit form
        def act(self):            # You probably want to be more explicit here
            A.act(self)           # This handles both single and multiple inher.
            B.act(self)           # So why use the super special case at all?

>>> C().act()
A
B

>>> class A:
        def act(self):
            print('A')
            super().act()         # Add a super here too

>>> class B:
        def act(self):
            print('B')
            super().act()         # Add a super here too
 
>>> class C(B, A):
        def act(self):
            super().act()         # Hope this winds up running all methods 
 
>>> C().act()
B
A
AttributeError: 'super' object has no attribute 'act'

>>> I = C()
>>> I.__class__.__mro__
(<class '__main__.C'>, <class '__main__.B'>, <class '__main__.A'>, <class 'object'>)

>>> class X:                          # Code a bogus class, just to appease super
        def act(self):
            print('anchor')

>>> class A: …same…
 
>>> class B: …same…

>>> class C(B, A, X):                 # Add a final anchor, just to appease super
         def act(self):
             super().act()

>>> C().act()
B
A
anchor

>>> [c.__name__ for c in C().__class__.__mro__]
['C', 'B', 'A', 'X', 'object']

>>> class X: …same…
         
>>> class A(X): …same…
 
>>> class B(X): …same…

>>> class C(B, A): …same…

>>> C().act()
B
A
anchor

>>> [c.__name__ for c in C().__class__.__mro__]
['C', 'B', 'A', 'X', 'object']

>>> class A:
        def __init__(self):
            print('A')
            super().__init__()       # Propagate constructor calls
 
>>> class B:
        def __init__(self):
            print('B')
            super().__init__()       # Propagate constructor calls
 
>>> class C(B, A):
        def __init__(self):
            super().__init__()       # Assume object anchors constructor chain

>>> I = C()
B
A

>>> class A:
        def __init__(self, name):         # Add an argument to the method
            print('A')
            super().__init__(name)
 
>>> class B:
        def __init__(self, name):
            print('B')
            super().__init__(name)
 
>>> class C(B, A):
        def __init__(self, name):
            super().__init__(name)        # But object's arguments list differs
 
>>> I = C('Pat')
B
A
TypeError: object.__init__() takes exactly one argument (the instance to initialize)

>>> class C:
        attr1 = 'hack'          # super also fetches data attributes
        def attr2(self):        # And returns bound methods sans calls
            return 'code'
 
>>> class D(C):
        def act(self):
            return super().attr1, super().attr2

>>> I = D()
>>> I.act()
('hack', <bound method C.attr2 of <__main__.D object at 0x103993200>>)
>>> I.act()[1]()
'code'

>>> class C:
        def __getitem__(self, ix):      # Indexing overload method
            print('C index')

>>> class D(C):
        def __getitem__(self, ix):      # Redefine to extend here
            print('D index')
            C.__getitem__(self, ix)     # Explicit call form works
            super().__getitem__(ix)     # Direct name calls work too
            super()[ix]                 # But operators do not! (__getattribute__)

>>> I = C()
>>> I[99]
C index
>>> I = D()
>>> I[99]
D index
C index
C index
TypeError: 'super' object is not subscriptable

The super Wrap-Up

So there you have it: a brief tutorial on the super built-in, for which you can find copious supplements in all the standard places, some of which seem as focused on defending super as on documenting it. Hopefully, the coverage here has given you a balanced view of this tool’s trade-offs while introducing its fundamentals.

As we’ve just seen, in single-inheritance class trees, the super call may be used to refer to parent superclasses generically without naming them explicitly. In multiple-inheritance trees, this call can also be used to implement cooperative method dispatch that propagates calls through a tree. The latter role may be especially useful in diamonds, as a conforming method call chain visits each superclass just once.

While these are clear upsides in some contexts, it’s important to know that super can also yield highly implicit behavior, which for some programs may not invoke superclasses as expected or required.

To summarize, the super method-dispatch technique generally imposes three main coding requirements:

• Anchors: the method called by super must exist—which requires extra code and calls if no call-chain anchor is present, and mix-in classes can’t be specialized for a single tree’s context.

• Arguments: the method called by super must have the same argument signature across the entire class tree—which can impair flexibility, especially for implementation-level methods like constructors.

• Deployment: every appearance of the method called by super but the last must use super itself—which can make it difficult to use existing code, change call ordering, override methods, and code self-contained classes.

In addition, super builds upon the already complex MRO, can mask problems when single-inheritance trees become multiple-inheritance trees, may select a class other than a superclass in multiple-inheritance trees, and constitutes yet another special case for attribute inheritance, which we’ll revisit in Chapter 40.

In the end, super is easy to use and relatively harmless in single-inheritance roles, but its unusual semantics, rigid requirements, and questionable net reward make it a mixed bag. Python programmers, especially those learning Python anew, might be better served by the more general and transparent coding paradigm of explicit class-name references.

But you should judge all this for yourself in an OOP Python program near you.

Changing Class Attributes Can Have Side Effects

Theoretically speaking, classes (and class instances) are mutable objects. As with built-in lists and dictionaries, you can change them in place by assigning to their attributes—and as with lists and dictionaries, this means that changing a class or instance object may impact multiple references to it.

That’s usually what we want and is how objects change their state in general, but awareness of this issue becomes especially critical when changing class attributes. Because all instances generated from a class share the class’s namespace, any changes at the class level are reflected in all instances unless they have their own versions of the changed class attributes.

In Python, we can normally change any attribute in any object to which we have a reference. Consider the following class. Inside the class body, the assignment to the name a generates an attribute X.a, which lives in the class object at runtime and will be inherited by all of X’s instances:

>>> class X:
        a = 1       # Class attribute

>>> I = X()
>>> I.a             # Inherited by instance
1
>>> X.a             # Accessible through class
1

So far, so good—this is the normal case. But notice what happens when we change the class attribute dynamically outside the class statement: it also changes the attribute in every object that inherits from the class. Moreover, new instances created from the class during this session or program run also get the dynamically set value, regardless of what the class’s source code says:

>>> X.a = 2         # May change more than X
>>> I.a             # I changes too
2
>>> J = X()         # J inherits from X's runtime values
>>> J.a             # (but assigning to J.a changes a in J, not X or I)
2

Is this a useful feature or a dangerous trap? You be the judge. As discussed in Chapter 27, you can actually get work done by changing class attributes without ever making a single instance—a technique that can simulate the use of “records” or “structs” in other languages. As a refresher, consider the following unusual but legal Python program:

class X: pass                       # Make a few attribute namespaces
class Y: pass

X.a = 1                             # Use class attributes as variables
X.b = 2                             # No instances anywhere to be found
X.c = 3
Y.a = X.a + X.b + X.c

for X.i in range(Y.a): print(X.i)   # Prints 0..5

Here, the classes X and Y work like “fileless” modules—namespaces for storing variables we don’t want to clash. This is a perfectly legal Python programming trick, but it’s less appropriate when applied to classes written by others; you can’t always be sure that class attributes you change aren’t critical to the class’s internal behavior. If you’re out to simulate a C struct, you may be better off changing instances than classes, as that way, only one object is affected:

class Record: pass
X = Record()
X.name = 'pat'
X.job  = 'Pizza maker'

Changing Mutable Class Attributes Can Have Side Effects, Too

This gotcha is really an extension of the prior. Because class attributes are shared by all instances, if a class attribute references a mutable object, changing that object in place from any instance impacts all instances at once:

>>> class C:
        shared = []                 # Class attribute
        def __init__(self):
            self.perobj = []        # Instance attribute

>>> x, y = C(), C()                 # Two instances
>>> y.shared, y.perobj              # Implicitly share class attrs
([], [])

>>> x.shared.append('hack')         # Impacts y's view too!
>>> x.perobj.append('code')         # Impacts x's data only
>>> x.shared, x.perobj
(['hack'], ['code']) 

>>> y.shared, y.perobj              # y sees change made through x
(['hack'], [])
>>> C.shared                        # Stored on class and shared
['hack']

This effect is no different than many we’ve seen in this book already: mutable objects are shared by simple variables, globals are shared by functions, module-level objects are shared by multiple importers, and mutable function arguments are shared by the caller and the callee. All of these are cases of general behavior—multiple references to a mutable object—and all are impacted if the shared object is changed in place from any reference.

Here, this occurs in class attributes shared by all instances via inheritance, but it’s the same phenomenon at work. It may be made more subtle by the different behavior of assignments to instance attributes themselves:

x.shared.append('hack')    # Changes shared object attached to class - in place
x.shared = 'hack'          # Changed or creates instance attribute attached to x

But again, this is not a problem, it’s just something to be aware of; shared mutable class attributes can have many valid uses in Python programs.

Multiple Inheritance: Order Matters

This may be obvious by now, but it’s worth underscoring one last time: if you use multiple inheritance, the order in which superclasses are listed in the class statement header can be critical. Python always searches superclasses from left to right, according to their order in the header line.

For instance, in the multiple inheritance example we studied in Chapter 31, imagine that the Super class implemented a __str__ method, too:

class ListTree:
    def __str__(self): …

class Super:
    def __str__(self): …

class Sub(ListTree, Super):    # Get ListTree's __str__ by listing it first

x = Sub()                      # Inheritance searches ListTree before Super

Which class would we inherit it from—ListTree or Super? As inheritance searches generally proceed from left to right, we would get the method from whichever class is listed first (leftmost) in Sub’s class header. Presumably, we would list ListTree first because its whole purpose is its custom __str__. Indeed, we had to do this in Chapter 31 when mixing this class with a tkinter.Button that had a __str__ of its own.

But now, suppose Super and ListTree have their own versions of other same-named attributes, too. If we want one name from Super and another from ListTree, the order in which we list them in the class header won’t help—we will have to override inheritance by manually assigning to the attribute name in the Sub class:

class ListTree:
    def __str__(self): …
    def other(self): …

class Super:
    def __str__(self): …
    def other(self): …

class Sub(ListTree, Super):    # Get ListTree's __str__ by listing it first
    other = Super.other        # But explicitly pick Super's version of other
    def __init__(self):
        …

x = Sub()                      # Inheritance searches Sub before ListTree/Super

Here, the assignment to other within the Sub class creates Sub.other—a reference back to the Super.other object. Because it is lower in the tree, Sub.other effectively hides ListTree.other, the attribute that the inheritance search would normally find. Similarly, if we listed Super first in the class header to pick up its other, we would need to select ListTree’s method explicitly:

class Sub(Super, ListTree):               # Get Super's other by order
    __str__ = ListTree.__str__            # Explicitly pick ListTree.__str__

For another example of the technique shown here in action, see the discussion of explicit conflict resolution in “Attribute Conflict Resolution” in Chapter 31. Ultimately, multiple inheritance is an advanced tool. Even if you understood the last paragraph, it’s still a good idea to use it sparingly and carefully. Otherwise, the meaning of a name may come to depend on the order in which classes are mixed in an arbitrarily far-removed subclass.

As a rule of thumb, multiple inheritance works best when your mix-in classes are as self-contained as possible—because they may be used in a variety of contexts, they should not make assumptions about names related to other classes in a tree. The pseudoprivate __X attributes feature we studied in Chapter 31 can help by localizing names that a class relies on owning and limiting the names that mix-in classes add to the mix. In this example, for instance, if ListTree only means to export its custom __str__, it can name its other method __other to avoid clashing with like-named classes in the tree.

Scopes in Methods and Classes

When working out the meaning of names in class-based code, it helps to remember that classes introduce local scopes, just as functions do, and methods are simply further nested functions. In the following example, the generate function returns an instance of the nested Hack class. Within its code, the class name Hack is assigned in the generate function’s local scope and hence is visible to any further nested functions, including code inside method; it’s in the E enclosing-function layer of the LEGB scope lookup rule:

def generate():
    class Hack:                  # Hack is a name in generate's local scope
        count = 1
        def method(self):
            print(Hack.count)    # Visible in generate's scope, per LEGB rule (E)
    return Hack()

generate().method()

This example works because the local scopes of all enclosing function defs are automatically visible to nested defs—including nested method defs, as in this example.

Even so, keep in mind that method defs cannot see the local scope of the enclosing class; they can see only the local scopes of enclosing defs. That’s why methods must go through the self instance or the class name to reference methods and other attributes defined in the enclosing class statement. For example, code in the method must use self.count or Hack.count, not just count.

To avoid nesting, we could restructure this code such that the class Hack is defined at the top level of the module: the nested method function and the top-level generate will then both find Hack in their global scopes; it’s not localized to a function’s scope, but is still local to a single module:

def generate():
    return Hack()

class Hack:                    # Define at top level of module
    count = 1
    def method(self):
        print(Hack.count)      # Found in global scope (enclosing module)

generate().method()

Code tends to be simpler in general if you avoid nesting classes and functions. On the other hand, class nesting is useful in closure contexts, where the enclosing function’s scope retains state used by the class or its methods. In the following, the nested method has access to its own scope, the enclosing function’s scope (for label), the enclosing module’s global scope, anything saved in the self instance by the class, and the class itself via its nonlocal name:

>>> def generate(label):       # Returns a class instead of an instance
        class Hack:
            count = 1
            def method(self):
                print(f'{label}={Hack.count}')
        return Hack

>>> aclass = generate('Gotchas')
>>> I = aclass()
>>> I.method()
Gotchas=1

Miscellaneous Class Gotchas

Here’s a handful of additional class-related warnings, mostly as review.

“Overwrapping-itis”

Finally, when used well, the code reuse features of OOP make it excel at cutting development time. Sometimes, though, OOP’s abstraction potential can be abused to the point of making code difficult to understand. If classes are layered too deeply, code can become obscure; you may have to search through many classes to discover what an operation does.

Imagine, for example, a framework with hundreds of classes and a dozen levels of inheritance (this is a true story, but details have been omitted to protect the innocent). Deciphering method calls in such a complex system may be a monumental task: multiple classes might have to be consulted for even the most basic of operations. In fact, the logic of such a system can be so deeply wrapped that understanding a piece of code in some cases may require days of wading through related files. This obviously isn’t ideal for programmer productivity.

The most general rule of thumb of Python programming applies here, too: don’t make things complicated unless they truly must be. Wrapping your code in multiple layers of classes to the point of incomprehensibility is always a bad idea. Abstraction is the basis of polymorphism and encapsulation, and it can be a very effective tool when used well. However, you’ll simplify debugging and aid maintainability if you make your class interfaces intuitive, avoid making your code overly abstract, and keep your class hierarchies short and small unless there is a good reason to do otherwise. Remember: code you write is generally code that others must read.