Constructors and Expressions: __init__ and __sub__

As a warm-up, consider the simple class in Example 30-1: its Number class, coded in module file number.py, provides a method to intercept instance construction (__init__), as well as one for catching subtraction expressions (__sub__). Special methods like these are the hooks that let you tie into built-in operations.

class Number:
    def __init__(self, start):                  # On Number(start)
        self.data = start
    def __sub__(self, other):                   # On instance - other
        return Number(self.data - other)        # Result is a new instance

As we’ve already learned, the __init__ constructor method seen in this code is the most commonly used operator-overloading method in Python; it’s present in most classes and used to initialize the newly created instance object using any arguments passed to the class name. The __sub__ method plays the binary-operator role that __add__ did in Chapter 27’s introduction, intercepting subtraction expressions and returning a new instance of the class as its result (and running __init__ along the way):

>>> from number import Number                   # Fetch class from module
>>> X = Number(5)                               # Number.__init__(X, 5)
>>> Y = X - 2                                   # Number.__sub__(X, 2)
>>> Y.data                                      # Y is new Number instance
3

We’ve already studied __init__ and basic binary operators like __sub__ in some depth, so we won’t rehash their usage further here. In this chapter, we will tour some other tools available in this domain and look at example code that applies them in common use cases.

Common Operator-Overloading Methods

Just about everything you can do to built-in objects such as integers and lists has a corresponding specially named method for overloading in classes. Table 30-1 lists a few of the most common; there are many more. In fact, many overloading methods come in multiple versions (e.g., __add__, __radd__, and __iadd__ for addition), which is one reason there are so many. See the Python language reference manual for an exhaustive list of the special method names available.

All overloading methods have names that start and end with two underscores to keep them distinct from other names you define in your classes. The mappings from special method names to expressions or operations are predefined by the Python language and documented in full in its standard language manual. For example, the name __add__ always maps to + expressions by Python language definition, regardless of what an __add__ method’s code actually does; it’s largely just a dispatch mechanism.

Operator-overloading methods may be inherited from superclasses if not defined, just like any other methods. Operator-overloading methods are also all optional—if you don’t code or inherit one, that operation is simply unsupported by your class, and attempting it will raise an exception. Some built-in operations, like printing, have defaults inherited from the implied object root class, but most built-ins fail for class instances if no corresponding operator-overloading method is present.

As we go along here, keep in mind that most overloading methods are used only in advanced programs that require objects to behave like built-ins, though the __init__ constructor we’ve already met tends to appear in most classes. With that qualifier, let’s explore some of the additional methods in Table 30-1 by example.

$ python3 -m timeit -n 10000 -r 10 \
             -s "L = list(range(100))" "x = L.__len__()"
10000 loops, best of 10: 53.4 nsec per loop

$ python3 -m timeit -n 10000 -r 10 \
             -s "L = list(range(100))" "x = len(L)"
10000 loops, best of 10: 25.3 nsec per loop

Intercepting Slices

Surprisingly, in addition to indexing, __getitem__ is also called for slice expressions. Formally speaking, built-in object types handle slicing the same way. For example, the following demos slicing at work on a built-in list, using upper and lower bounds, omitted parts, and a stride (see Chapter 7 if you need a refresher on slicing):

>>> L = [5, 6, 7, 8, 9]
>>> L[2:4]                              # Slice with slice syntax: 2..(4-1)
[7, 8]
>>> L[1:]
[6, 7, 8, 9]
>>> L[:-1]
[5, 6, 7, 8]
>>> L[::2]
[5, 7, 9]

Really, though, slicing bounds are bundled up into a slice object and passed to the list’s implementation of indexing. In fact, you can always pass a slice object manually—slice syntax is mostly syntactic sugar for indexing with a slice object:

>>> L[slice(2, 4)]                      # Slice with slice objects
[7, 8]
>>> L[slice(1, None)]
[6, 7, 8, 9]
>>> L[slice(None, -1)]
[5, 6, 7, 8]
>>> L[slice(None, None, 2)]
[5, 7, 9]

This matters in classes with a __getitem__ method—this method will be called both for basic indexing (with an index or key) and for slicing (with a slice object). Our previous class won’t handle slicing because its math assumes integer indexes are passed, but the following class will. When called for indexing, the argument is an integer as before:

>>> class Indexer:
        def __init__(self, data):
            self.data = data
        def __getitem__(self, index):    # Called for index or slice
            print('getitem:', index)
            return self.data[index]      # Perform index or slice

>>> X = Indexer([5, 6, 7, 8, 9])
>>> X[0]                                 # Indexing sends __getitem__ an integer
getitem: 0
5
>>> X[1]
getitem: 1
6
>>> X[-1]
getitem: −1
9

When called for slicing, though, the method receives a slice object, which is simply passed along to the embedded list indexer in a new index expression:

>>> X[2:4]                             # Slicing sends __getitem__ a slice object
getitem: slice(2, 4, None)
[7, 8]
>>> X[1:]
getitem: slice(1, None, None)
[6, 7, 8, 9]
>>> X[:-1]
getitem: slice(None, −1, None)
[5, 6, 7, 8]
>>> X[::2]
getitem: slice(None, None, 2)
[5, 7, 9]

Where needed, __getitem__ can test the type of its argument, and extract slice object bounds—slice objects have attributes start, stop, and step, any of which can be None if omitted:

>>> class Indexer:
        def __getitem__(self, index):
            if isinstance(index, int):               # Test usage mode
                print('indexing', index)
            else:
                print('slicing', index.start, index.stop, index.step)

>>> X = Indexer()
>>> X[99]
indexing 99
>>> X[1:99:2]
slicing 1 99 2
>>> X[1:]
slicing 1 None None

Run a help(slice) in a REPL for more info on this very special-case built-in object type, and see the section “Membership: __contains__, __iter__, and __getitem__” for another example of slice interception at work.

Intercepting Item Assignments

If used, the __setitem__ index assignment method similarly intercepts both index and slice assignments—it receives a slice object for the latter, which may be passed along in another index assignment or used directly in the same way:

>>> class IndexSetter:
        def __init__(self, data):
            self.data = data
        def __setitem__(self, index, value):    # Catch index or slice assignment
            print('setitem:', index)
            self.data[index] = value            # Assign index or slice

>>> X = IndexSetter([5, 6, 7, 8, 9])
>>> X[0] = 555
setitem: 0
>>> X[-2:] = [888, 999, 111]
setitem: slice(-2, None, None)

>>> X.data
[555, 6, 7, 888, 999, 111]

In fact, __getitem__ may be called automatically in even more contexts than indexing and slicing—it’s also an iteration fallback option, as you’ll see in a moment. First, though, let’s clear up a potential point of confusion in this category.

But __index__ Means As-Integer

Don’t mistake the (perhaps unfortunately named) __index__ method for __getitem__ index interception. The __index__ method returns an integer value for an instance when one is needed—and in retrospect, might have been better named __asindex__. For example, it’s used by built-ins that convert to digit strings:

>>> class C:
        def __index__(self):
            return 255

>>> X = C()
>>> hex(X)               # Integer value
'0xff'
>>> bin(X)
'0b11111111'
>>> oct(X)
'0o377'

Although this method does not intercept instance indexing like __getitem__, it is also used in contexts that require an integer—including indexing and slicing:

>>> eds = [f'LP{i}e' for i in range(256)]
>>> eds[255]
'LP255e'
>>> X = C()
>>> eds[X]               # As index (not X[i]!)
'LP255e'
>>> eds[X:]              # As index (not X[i:]!)
['LP255e']

Though arguably misnamed, there is a rich history of former methods that __index__ subsumes, which we’ll mercifully omit here. The __getitem__ method also subsumes former tools but retains a fallback role up next.

User-Defined Iterables

In the __iter__ scheme, classes implement user-defined iterables by simply implementing the iteration protocol introduced in Chapter 14 and elaborated in Chapter 20. For example, the code in Example 30-2 uses a class to define a user-defined iterable that generates squares on demand, instead of all at once.

class Squares:
    def __init__(self, start, stop):    # Save state when created
        self.value = start - 1
        self.stop  = stop

    def __iter__(self):                 # Return iterator object
        return self                     # Also called by iter() built-in

    def __next__(self):                 # Return a square on each iteration
        if self.value == self.stop:     # Also called by next() built-in
            raise StopIteration
        self.value += 1
        return self.value ** 2

When imported, its instances can appear in iteration tools just like built-ins:

$ python3
>>> from squares import Squares
>>> for i in Squares(1, 5):             # for calls __iter__
        print(i, end=' ')               # Each iteration calls __next__

1 4 9 16 25

Here, the iterator object returned by __iter__ is simply the instance self because the __next__ method is part of this class itself. In more complex scenarios, the iterator object may be defined as a separate class and object with its own state information to support multiple active iterations over the same instance data (we’ll code an example of this in a moment).

The end of the iteration is signaled with a Python raise statement—introduced in Chapter 29 and covered in full in the next part of this book, but which simply raises an exception as if Python itself had done so. Because of all this, manual iterations work the same on user-defined iterables as they do on built-in objects:

>>> X = Squares(1, 5)                   # Iterate manually: what loops do
>>> I = iter(X)                         # iter calls __iter__
>>> next(I)                             # next calls __next__
1
>>> next(I)
4
…more omitted…
>>> next(I)
25
>>> next(I)                             # Can catch this in try statement
StopIteration

An equivalent coding of this iterable with __getitem__ might be less natural because the for would then iterate through all offsets zero and higher; the offsets passed in would be only indirectly related to the range of values produced (0…N would need to map to start…stop). Because __iter__ objects retain explicitly managed state between next calls, they can be more general than __getitem__.

On the other hand, iterables based on __iter__ can sometimes be more complex and less functional than those based on __getitem__. They are really designed for iteration, not random indexing—in fact, they don’t overload the indexing expression at all, though you can collect their items in a sequence such as a list to enable other operations:

>>> X = Squares(1, 5)
>>> X[1]
TypeError: 'Squares' object is not subscriptable
>>> list(X)[1]
4

>>> X = Squares(1, 5)                   # Make an iterable with state
>>> [n for n in X]                      # Exhausts items: __iter__ returns self
[1, 4, 9, 16, 25]
>>> [n for n in X]                      # Now it's empty: __iter__ returns same self
[]

>>> [n for n in Squares(1, 5)]          # Make a new iterable object
[1, 4, 9, 16, 25]
>>> list(Squares(1, 3))                 # A new object for each new __iter__ call
[1, 4, 9]

>>> 36 in Squares(1, 10)                # Other iteration tools
True
>>> a, b, c = Squares(1, 3)             # Each calls __iter__ and then __next__
>>> a, b, c
(1, 4, 9)
>>> ':'.join(map(str, Squares(1, 5)))
'1:4:9:16:25'

>>> X = Squares(1, 5)
>>> tuple(X), tuple(X)                  # Iterator exhausted in second tuple()
((1, 4, 9, 16, 25), ())

>>> X = list(Squares(1, 5))
>>> tuple(X), tuple(X)
((1, 4, 9, 16, 25), (1, 4, 9, 16, 25))

>>> def gsquares(start, stop):                   # Generator function
        for i in range(start, stop + 1):
            yield i ** 2

>>> for i in gsquares(1, 5):
        print(i, end=' ')

1 4 9 16 25

>>> for i in (x ** 2 for x in range(1, 6)):      # Generator expression
        print(i, end=' ')

1 4 9 16 25

>>> [x ** 2 for x in range(1, 6)]
[1, 4, 9, 16, 25]

Multiple Iterators on One Object

Earlier, it was mentioned in passing that the iterator object (with a __next__) produced by an iterable may be defined as a separate class with its own state information to more directly support multiple active iterations over the same data. To understand this better, consider what happens when we step across a built-in type like a string:

>>> S = 'ace'
>>> for x in S:
        for y in S:
            print(x + y, end=' ')

aa ac ae ca cc ce ea ec ee

Here, the outer loop grabs an iterator from the string by calling iter, and each nested loop does the same to get an independent iterator. Because each active iterator has its own state information, each loop can maintain its own position in the string, regardless of any other active loops. Moreover, we’re not required to make a new string or convert to a list each time; the single string object itself supports multiple scans.

We saw related examples earlier, in Chapters 14 and 20. For instance, generator functions and expressions, as well as built-ins like map and zip, proved to be single-iterator objects, thus supporting a single active scan. By contrast, the range built-in, and other built-in types like lists, support multiple active iterators with independent positions.

When we code user-defined iterables with classes, it’s up to us to decide whether we will support a single active iteration or many. To achieve the multiple-iterator effect, __iter__ simply needs to define a new stateful object for the iterator instead of returning self for each iterator request.

For example, the class SkipObject in Example 30-3 defines an iterable object that skips every other item on iterations. Because its iterator object is created anew from a supplemental class for each iteration, it supports multiple active loops directly.

class SkipObject:
    def __init__(self, wrapped):                  # Save item to be used
        self.wrapped = wrapped

    def __iter__(self):
        return SkipIterator(self.wrapped)         # New iterator each time

class SkipIterator:
    def __init__(self, wrapped):
        self.wrapped = wrapped                    # Iterator state information
        self.offset  = 0

    def __next__(self):
        if self.offset >= len(self.wrapped):      # Terminate iterations
            raise StopIteration
        else:
            item = self.wrapped[self.offset]      # else return and skip
            self.offset += 2
            return item

if __name__ == '__main__':
    alpha = 'abcdef'
    skipper = SkipObject(alpha)                   # Make container object
    I = iter(skipper)                             # Make an iterator on it
    print(next(I), next(I), next(I))              # Visit offsets 0, 2, 4

    for x in skipper:                # for calls __iter__ automatically
        for y in skipper:            # Nested fors call __iter__ again each time
            print(x + y, end=' ')    # Each iterator has its own state, offset

When run, this example works like the earlier nested loops with built-in strings. Each active loop has its own position in the string because each obtains an independent iterator object that records its own state information:

$ python3 skipper.py
a c e
aa ac ae ca cc ce ea ec ee

By contrast, our earlier Squares class of Example 30-2 supports just one active iteration unless we call Squares again in nested loops to obtain new objects (else the outer for loop would run just once). Here, there is just one SkipObject iterable, with multiple iterator objects created from it.

>>> S = 'abcdef'
>>> for x in S[::2]:
        for y in S[::2]:             # New objects on each iteration
            print(x + y, end=' ')

aa ac ae ca cc ce ea ec ee

>>> S = 'abcdef'
>>> S = S[::2]
>>> S
'ace'
>>> for x in S:
        for y in S:                  # Same object, new iterators
            print(x + y, end=' ')

aa ac ae ca cc ce ea ec ee

Coding Alternative: __iter__ plus yield

Now for something more implicit—but potentially useful nonetheless. In some applications, it’s possible to minimize coding requirements for user-defined iterables by combining the __iter__ method we’re exploring here and the yield generator function statement we studied in Chapter 20. Because generator functions automatically save local-variable state and create required iterator methods, they fit this role well and complement the state retention and other utility we get from classes.

As a quick review, recall that any function that contains a yield statement is turned into a generator function. When called, it returns a new generator object with automatic retention of local scope and code position; an automatically created __iter__ method that simply returns itself; and an automatically created __next__ method that starts the function or resumes it where it last left off:

>>> def gen(x):
       for i in range(x): yield i ** 2

>>> G = gen(5)               # Create a generator with __iter__ and __next__
>>> G.__iter__() is G        # Both methods exist on the same object
True
>>> I = iter(G)              # Runs __iter__: generator returns itself
>>> next(I), next(I)         # Runs __next__
(0, 1)
>>> list(gen(5))             # Iteration tools automatically run iter and next
[0, 1, 4, 9, 16]

This is still true even if the generator function with a yield happens to be a method named __iter__ . Whenever invoked by an iteration tool, such a method will return a new generator object with the requisite __next__. As an added bonus, generator functions coded as methods in classes have access to saved state in both instance attributes and local-scope variables.

For example, the class in Example 30-4 is equivalent to the initial Squares user-defined iterable we coded earlier in Example 30-2, but noticeably shorter (4 lines, for anyone counting).

class Squares:                                 # __iter__ + yield generator
    def __init__(self, start, stop):           # __next__ is automatic/implied
        self.start = start
        self.stop  = stop

    def __iter__(self):
        for value in range(self.start, self.stop + 1):
            yield value ** 2

As before, for loops and other iteration tools iterate through instances of this class automatically:

$ python3
>>> from squares_yield import Squares
>>> for i in Squares(1, 5): print(i, end=' ')      # Runs __iter__, then __next__

1 4 9 16 25

And as always, we can also look under the hood to see how this actually works in iteration tools like for. Running our class instance through iter obtains the result of calling __iter__ as usual. In this case, though, the result is a generator object with an automatically created __next__ of the same sort we always get when calling a generator function that contains a yield. The only difference here is that the generator function is automatically called on iter. Invoking the result object’s next interface produces results on demand:

>>> S = Squares(1, 5)          # Runs __init__: class saves instance state
>>> S
<squares_yield.Squares object at 0x109e30b90> 

>>> I = iter(S)                # Runs __iter__: returns a generator
>>> I
<generator object Squares.__iter__ at 0x109ecb3e0>
>>> next(I)
1
>>> next(I)                    # Runs generator's __next__
4
…etc…
>>> next(I)                    # Generator has both instance and local scope state
StopIteration

It may also help to notice that we could name the generator method something other than __iter__ and call manually to iterate—Squares(…).gen(), for example. Using the __iter__ name invoked automatically by iteration tools simply skips a manual attribute fetch and call step. Example 30-5 demos the idea.

import squares_yield                      # Reuse prior example's __init__

class Squares(squares_yield.Squares):     # Non __iter__ equivalent 
    def gen(self):
        for value in range(self.start, self.stop + 1):
            yield value ** 2

The example also imports Example 30-4 to inherit its constructor. When run, its results are the same, but we must call the gen method explicitly to fetch an iterable—a step that __iter__ automates and obviates:

$ python3
>>> from squares_yield_manual import Squares
>>> for i in Squares(1, 5).gen(): print(i, end=' ')
…same results…

>>> S = Squares(1, 5)
>>> I = iter(S.gen())          # Call generator manually for iterable/iterator
>>> next(I)
…same results…

Coding the generator as __iter__ instead cuts out the middleperson in your code, though both schemes ultimately wind up creating a new generator object for each iteration:

• With __iter__, iteration triggers __iter__, which returns a new generator with __next__.

• Without __iter__, your code calls to make a generator, which returns itself for __iter__.

See Chapter 20 for more on yield and generators if this is puzzling, and compare it with the more explicit __next__ version in Example 30-2 earlier. If you do, you’ll notice that the squares_yield.py version is 4 lines shorter (7 versus 11, not counting whitespace). In a sense, this scheme reduces class coding requirements much like the closure functions of Chapter 17, but in this case does so with a combination of functional and OOP techniques instead of an alternative to classes. For example, the generator method still leverages self attributes.

This may also seem like one too many levels of magic to some observers—it relies on both the iteration protocol and the object creation of generators, both of which are highly implicit (in contradiction of longstanding Python goals). Opinions aside, it’s important to understand the non-yield flavor of class iterables too, because it’s explicit, general, and sometimes broader in scope.

Still, the __iter__/yield technique may prove effective in cases where it applies. It also comes with a substantial advantage—as the next section explains.

$ python3
>>> from squares_yield import Squares   # Using the __iter__/yield Squares
>>> S = Squares(1, 5)
>>> I = iter(S)
>>> next(I); next(I)
1
4
>>> K = iter(S)                         # With yield, multiple iterators automatic
>>> next(K)
1
>>> next(I)                             # I is independent of K: own local state
9

>>> S = Squares(1, 3)
>>> for i in S:                         # Each "for" calls __iter__
        for j in S:
            print(f'{i}:{j}', end=' ')

1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9

class Squares:
    def __init__(self, start, stop):                 # Non-yield generator
        self.start = start                           # Multiscans: extra object
        self.stop  = stop

    def __iter__(self):
        return SquaresIter(self.start, self.stop)

class SquaresIter:
    def __init__(self, start, stop):
        self.value = start - 1
        self.stop  = stop

    def __next__(self):
        if self.value == self.stop:
            raise StopIteration
        self.value += 1
        return self.value ** 2

$ python3
>>> from squares_nonyield import Squares
>>> for i in Squares(1, 5): print(i, end=' ')

1 4 9 16 25

>>> S = Squares(1, 5)
>>> I = iter(S)
>>> next(I); next(I)
1
4
>>> K = iter(S)                          # Multiple iterators without yield
>>> next(K)
1
>>> next(I)
9

>>> S = Squares(1, 3)
>>> for i in S:                          # Each "for" calls __iter__
        for j in S:
            print(f'{i}:{j}', end=' ')

1:1 1:4 1:9 4:1 4:4 4:9 9:1 9:4 9:9

class SkipObject:                           # Another __iter__ + yield generator
    def __init__(self, wrapped):            # Instance scope retained normally
        self.wrapped = wrapped              # Local scope state saved auto

    def __iter__(self):
        offset = 0
        while offset < len(self.wrapped):
            item = self.wrapped[offset]
            offset += 2
            yield item

$ python3
>>> from skipper_yield import SkipObject
>>> skipper = SkipObject('abcdef')
>>> I = iter(skipper)
>>> next(I); next(I); next(I)
'a'
'c'
'e'
>>> for x in skipper:               # Each "for" calls __iter__: new auto generator
        for y in skipper:
            print(x + y, end=' ')

aa ac ae ca cc ce ea ec ee

Attribute Reference

The __getattr__ method intercepts attribute references. It’s called with the attribute name as a string whenever you try to qualify an instance with an undefined (nonexistent) attribute name. It is not called if Python can find the attribute using its inheritance tree search procedure.

Because of its behavior, __getattr__ is useful as a hook for responding to attribute requests in a generic fashion. It’s commonly used to delegate calls to embedded (or “wrapped”) objects from a proxy controller object—of the sort introduced in Chapter 28’s introduction to delegation. This method can also be used to adapt classes to an interface or add accessors for data attributes after the fact—logic in a method that validates or computes an attribute after it’s already being used with simple dot notation (possibly after a rename of the original).

The basic mechanism underlying these goals is straightforward—the following class catches attribute references, computing the value for one dynamically, and triggering an error for others unsupported with the raise statement described earlier in this chapter for iterators (and again, fully covered in Part VII):

>>> class Empty:
        def __getattr__(self, attrname):           # On self.undefined
            if attrname == 'age':
                return 40
            else:
                raise AttributeError(attrname)

>>> X = Empty()
>>> X.age                  # Becomes X.__getattr__('age')
40
>>> X.name                 # Unsupported attribute
…error text omitted…
AttributeError: name

Here, the Empty class and its instance X have no real attributes of their own, so the access to X.age gets routed to the __getattr__ method; self is assigned the instance (X), and attrname is assigned the undefined attribute name string ('age'). The class makes age look like a real attribute by returning a real value as the result of the X.age qualification expression (40). In effect, age becomes a dynamically computed attribute—its value is formed by running code, not fetching an object.

For attributes that the class doesn’t know how to handle, __getattr__ raises the built-in AttributeError exception to tell Python that these are bona fide undefined names; asking for X.name triggers the error. You’ll see __getattr__ again when explore delegation and properties at work in the next two chapters; let’s move on to related tools here.

Attribute Assignment and Deletion

In the same department, the __setattr__ intercepts all attribute assignments. If this method is defined or inherited, self.attr = value becomes self.__setattr__('attr', value). Like __getattr__, this allows your class to catch attribute changes and validate or transform as desired.

This method is a bit trickier to use, though, because assigning to any self attributes within __setattr__ calls __setattr__ again, potentially causing an infinite recursion loop (and a fairly quick stack overflow exception!). In fact, this applies to all self attribute assignments anywhere in the class—all are routed to __setattr__, even those in other methods, and those to names other than that which may have triggered __setattr__ in the first place. So be warned: this catches all attribute assignments.

If you wish to use this method, you can avoid loops by coding instance attribute assignments as assignments to attribute dictionary keys. That is, use self.__dict__['name'] = x, not self.name = x; because you’re not assigning to __dict__ itself, this avoids the loop:

>>> class Accesscontrol:
        def __setattr__(self, attr, value):
            if attr == 'age':
                self.__dict__[attr] = value + 10      # Not self.name=val or setattr
            else:
                raise AttributeError(attr + ' not allowed')

>>> X = Accesscontrol()
>>> X.age = 40                # Becomes X.__setattr__('age', 40)
>>> X.age                     # Found in __dict__ as usual
50
>>> X.name = 'Pat'            # Unsupported attribute
…text omitted…
AttributeError: name not allowed

If you change the __dict__ assignment within this class to either of the following, it triggers the infinite recursion loop and exception—both dot notation and its setattr built-in function equivalent (the assignment analog of getattr) fail when age is assigned outside the class:

self.age = value + 10                            # Loops!
setattr(self, attr, value + 10)                  # Loops! (attr is 'age')

An assignment to any other self attribute within the class triggers a recursive __setattr__ call too, though in this class ends less dramatically in the manual AttributeError exception:

self.other = 99                                  # Recurs + fails, but doesn't loop

It’s also possible to avoid recursive loops in a class that uses __setattr__ by rerouting any attribute assignments to a higher superclass with a call, instead of assigning keys in __dict__:

object.__setattr__(self, attr, value + 10)       # OK: doesn't loop (preview)

This uses an explicit-class call to the implied object superclass above all topmost classes (and has a super().__setattr__(…) equivalent sans self per Chapter 28’s “The super Alternative”). In fact, this alternative may be required in some classes per the upcoming note, though this is rare in practice.

A third attribute-management method, __delattr__, is passed the attribute name string and invoked on all attribute deletions (i.e., del object.attr). Like __setattr__, it must avoid recursive loops by running attribute deletions within the class using __dict__, or rerouting them to a superclass.

Other Attribute-Management Tools

The three attribute-access overloading methods we’ve met so far allow you to control or specialize access to attributes in your objects. They tend to play highly specialized roles, some of which we’ll explore later in this book. For another example of __getattr__ at work, see Chapter 28’s person-composite.py (Example 28-11).

And for future reference, keep in mind that there are other ways to manage attribute access in Python:

• The __getattribute__ method intercepts all attribute fetches, not just those that are undefined; like __setattr__, it must avoid loops for other attribute fetches in the class, usually with object rerouting.

• The property built-in function allows us to associate methods with fetch and set operations on a specific class attribute; it cannot catch accesses generically but can define what some do.

• Descriptors provide a protocol for associating __get__ and __set__ methods of a class with accesses to a specific instance or class attribute; they are as focused as property and, in fact, are used to implement it.

• Slots attributes are declared in classes but create implicit storage in each instance; if present, generic tools may need to list, fetch, and assign with schemes described in the preceding note.

Because these are all advanced tools that are not of interest to every Python programmer, we’ll defer the details of other attribute-management techniques until Chapter 32, and await their focused coverage in the online-only “Managed Attributes” chapter.

Emulating Privacy for Instance Attributes: Part 1

As another use case for attribute tools, the code in Example 30-10—file private0.py—generalizes the previous example, to allow each subclass to have its own list of private names that cannot be assigned to its instances (and raises a built-in exception with raise, which you’ll have to take on faith until Part VII).

class Privacy:
    def __setattr__(self, attr, value):             # On self.attr = value
        if attr in self.privates:
            raise NameError(f'{attr!r} for {self}') 
        else:
            self.__dict__[attr] = value             # Avoid loops by using dict key

class Test1(Privacy):
    privates = ['age']

class Test2(Privacy):
    privates = ['name', 'pay']
    def __init__(self):
        self.__dict__['name'] = 'Pat'               # To do better, see the 
                                                    # online-only "Decorators" chapter

if __name__ == '__main__':
    x = Test1()
    x.name = 'Sue'      # Works
    print(x.name)
   #x.age = 40          # Fails

    y = Test2()
    y.age = 30          # Works
    print(y.age)
   #y.name = 'Bob'      # Fails

This is a first-cut solution for an implementation of attribute privacy in Python—disallowing changes to attribute names outside a class. Although Python doesn’t support private declarations per se, techniques like this can emulate much of their purpose.

This particular attempt, though, is a partial—and even clumsy—solution. To make it more effective, we must augment it to allow classes to set their private attributes more naturally, without having to go through __dict__ each time, as the constructor must do here to avoid triggering __setattr__ and an exception. A better and more complete approach might require a wrapper (“proxy”) class to check for private attribute accesses made outside the class only and a __getattr__ to validate attribute fetches too.

We’ll postpone a more complete solution to attribute privacy until the online-only “Decorators” chapter, where we’ll use class decorators to intercept and validate attributes more generally. Even though privacy can be emulated this way, though, it almost never is in practice. Python programmers are able to write large OOP frameworks and applications without private declarations—an interesting finding about access controls in general that is beyond the scope of our purposes here.

Still, catching attribute references and assignments is generally a useful technique; it supports delegation, a design technique that allows controller objects to wrap up embedded objects, add new behaviors, and route other operations back to the wrapped objects. Because they involve design topics, we’ll revisit delegation and wrapper classes in the next chapter. Here, it’s time to move ahead in the operator overloading domain.

Why Two Display Methods?

So far, this section has largely been review. But while these methods are generally straightforward to use, their roles and behavior have some subtle implications for both design and coding. In particular, Python provides its two display methods to support alternative displays for different audiences:

• __str__ is tried first for the print operation and the str built-in function (the internal equivalent of which print runs). It generally should return a user-friendly display.

• __repr__ is used in all other contexts: for interactive echoes, the repr function, and nested appearances, as well as by print and str if no __str__ is present. It should generally return an as-code string that could be used to re-create the object or a detailed display for developers.

That is, __repr__ is used everywhere, except by print and str when a __str__ is defined. This means you can code a __repr__ to define a single display format used everywhere and may code a __str__ to either support print and str exclusively or to provide an alternative display for them.

General tools may also prefer __str__ to leave other classes the option of adding an alternative __repr__ display for use in other contexts, as long as print and str displays suffice for the tool. Conversely, a general tool that codes a __repr__ still leaves clients the option of adding alternative displays with a __str__ for print and str. In other words, if you code either, the other is available for an additional display. In cases where the choice isn’t clear, use __str__ for higher-level displays and __repr__ for lower-level displays and all-inclusive roles.

Let’s write some code to illustrate these two methods’ distinctions in more concrete terms. The prior example in this section showed how __repr__ is used as the fallback option in many contexts. However, while printing falls back on __repr__ if no __str__ is defined, the inverse is not true—other contexts, such as interactive echoes, use __repr__ only and don’t try __str__ at all:

>>> class addstr(adder):
        def __str__(self):                       # __str__ but no __repr__
            return f'[Value: {self.data}]'       # Convert to nice string

>>> x = addstr(3)
>>> x + 1
>>> x                                            # Default __repr__ (in object)
<__main__.addstr object at 0x106111b20> 
>>> print(x)                                     # Runs __str__ (in addstr)
[Value: 4]
>>> str(x), repr(x)
('[Value: 4]', '<__main__.addstr object at 0x106111b20>')

Because of this, __repr__ may be best if you want a single display for all contexts. By defining both methods, though, you can support different displays in different contexts—for example, a class’s end-user display with __str__, and a class’s developer display with __repr__. In effect, __str__ simply overrides __repr__ for more user-friendly display contexts:

>>> class addboth(adder):
        def __str__(self):
            return f'[Value: {self.data}]'       # User-friendly string
        def __repr__(self):
            return f'addboth({self.data})'       # As-code string

>>> x = addboth(4)
>>> x + 1
>>> x                                            # Runs __repr__
addboth(5)
>>> print(x)                                     # Runs __str__
[Value: 5]
>>> str(x), repr(x)
('[Value: 5]', 'addboth(5)')

Bonus: your classes’ __str__ and __repr__ are also run automatically by all three string-formatting tools of Chapter 7. This may not be a shocker, given these tools are defined to work like str and repr in this role, but display overloading is not just about REPL echoes and print:

>>> f'{x!s} {x!r}',  '{!s} {!r}'.format(x, x),  '%s %r' % (x, x)
('[Value: 5] addboth(5)', '[Value: 5] addboth(5)', '[Value: 5] addboth(5)')

Display Usage Notes

Though generally simple to use, three points regarding these display methods are worth calling out here. First, keep in mind that __str__ and __repr__ must both return strings; other result types are not converted and raise errors, so be sure to run them through a to-string converter (e.g., str or f'…') if needed.

Second, depending on a container’s string-conversion logic, the user-friendly display of __str__ might only apply when objects appear at the top level of a print operation; objects nested in larger objects might still print with their __repr__ or its default. The following illustrates both of these points:

>>> class Printer:
        def __init__(self, val):
            self.val = val
        def __str__(self):                  # Used for instance itself
            return str(self.val)            # Convert to a string result

>>> objs = [Printer(2), Printer(3)]
>>> for x in objs: print(x)                 # __str__ run when instance printed
                                            # But not when instance is in a list!
2
3
>>> print(objs)
[<__main__.Printer object at 0x106110c80>, <__main__.Printer object at 0x1060d2570>]
>>> objs
[<__main__.Printer object at 0x106110c80>, <__main__.Printer object at 0x1060d2570>]

To ensure that a custom display is run in all contexts regardless of the container, code __repr__, not __str__; the former is run in all cases if the latter doesn’t apply, including nested appearances:

>>> class Printer:
        def __init__(self, val):
            self.val = val
        def __repr__(self):                 # __repr__ used by print if no __str__
            return str(self.val)            # __repr__ used if echoed or nested

>>> objs = [Printer(2), Printer(3)]
>>> for x in objs: print(x)                 # No __str__: runs __repr__

2
3
>>> print(objs)                             # Runs __repr__, not __str__
[2, 3]
>>> objs
[2, 3]

Third, and perhaps most subtle, the display methods also have the potential to trigger infinite recursion loops in rare contexts—because some objects’ displays include displays of other objects, it’s not impossible that a display may trigger a display of an object being displayed, and thus loop. This is rare and obscure enough to skip here but watch for an example of this looping potential to appear for these methods in a note near the end of the next chapter about its listinherited.py class of Example 31-12, where __repr__ can loop.

In practice, __str__, and its more inclusive relative __repr__, seem to be the second most commonly used operator-overloading methods in Python scripts, behind __init__. Anytime you can print an object and see a custom display, one of these two tools is probably in use. For additional examples of these tools at work and the design trade-offs they imply, see Chapter 28’s case study and Chapter 31’s class-lister mix-ins, as well as their role in Chapter 35’s exception classes, where __str__ is required over __repr__.

Right-Side Addition

For instance, the __add__ methods coded so far technically do not support the use of instance objects on the right side of the + operator:

>>> class Adder:
       def __init__(self, value=0):
           self.data = value
       def __add__(self, other):
           return self.data + other

>>> x = Adder(5)
>>> x + 2
7
>>> 2 + x
TypeError: unsupported operand type(s) for +: 'int' and 'Adder'

To implement more general expressions and hence support commutative-style operators, code the __radd__ method as well. Python calls __radd__ only when the object on the right side of the + is your class instance, but the object on the left is not an instance of your class. The __add__ method for the object on the left is called instead in all other cases. As a demo, consider the class in Example 30-11 (which we’ll be adding to in a moment).

class Commuter1:
    def __init__(self, val):
        self.val = val

    def __add__(self, other):
        print('add', self.val, other)
        return self.val + other

    def __radd__(self, other):
        print('radd', self.val, other)
        return other + self.val

Because this defines both left- and right-side overloads for +, instances can appear on either side, or both:

>>> from commuter import Commuter1
>>> x = Commuter1(88)
>>> y = Commuter1(99)

>>> x + 1                      # __add__: instance + noninstance
add 88 1
89
>>> 1 + y                      # __radd__: noninstance + instance
radd 99 1
100
>>> x + y                      # __add__: instance + instance => triggers __radd__
add 88 <commuter.Commuter1 object at 0x1011b63f0>
radd 99 88
187

Notice how the order is reversed in __radd__: self is really on the right of the +, and other is on the left. Also note that x and y are instances of the same class here; when instances of different classes appear mixed in an expression, Python prefers the class of the one on the left. When we add the two instances of this class together, Python runs __add__, which in turn triggers __radd__ by simplifying the left operand and re-adding.

class Commuter2:
    def __init__(self, val):
        self.val = val

    def __add__(self, other):
        print('add', self.val, other)
        return self.val + other

    def __radd__(self, other):
        return self.__add__(other)              # Call __add__ explicitly

class Commuter3:
    def __init__(self, val):
        self.val = val

    def __add__(self, other):
        print('add', self.val, other)
        return self.val + other

    def __radd__(self, other):
        return self + other                     # Swap order and re-add

class Commuter4:
    def __init__(self, val):
        self.val = val

    def __add__(self, other):
        print('add', self.val, other)
        return self.val + other

    __radd__ = __add__                          # Alias: cut out the middleperson

class Commuter5:                                # Propagate class type in results
    def __init__(self, val):
        self.val = val

    def __add__(self, other):
        if isinstance(other, Commuter5):        # Type test to avoid object nesting
            other = other.val
        return Commuter5(self.val + other)      # Else + result is another Commuter

    def __radd__(self, other):
        return Commuter5(other + self.val)

    def __repr__(self):
        return f'Commuter5({self.val})'

>>> from commuter import Commuter5
>>> x = Commuter5(88)
>>> y = Commuter5(99)
>>> x
Commuter5(88)
>>> x + 1                      # Result is another Commuter instance
Commuter5(89)
>>> 1 + y
Commuter5(100) 

>>> z = x + y                  # Not nested: doesn't recur to __radd__
>>> z
Commuter5(187)
>>> z + 10
Commuter5(197)
>>> z + z
Commuter5(374)
>>> z + z + 1
Commuter5(375)

>>> z = x + y                  # With isinstance test+action commented-out       
>>> z
Commuter5(Commuter5(187))
>>> z + 10
Commuter5(Commuter5(197))
>>> z + z
Commuter5(Commuter5(Commuter5(Commuter5(374))))
>>> z + z + 1
Commuter5(Commuter5(Commuter5(Commuter5(375))))

class Commuter6:                                # Propagate class type in results
    def __init__(self, val):
        if isinstance(val, Commuter6):          # Type test to avoid object nesting
            self.val = val.val
        else:
            self.val = val

    def __add__(self, other):
        return Commuter6(self.val + other)

    def __radd__(self, other):
        return Commuter6(other + self.val)

    def __repr__(self):
        return f'Commuter6({self.val})'

if __name__ == '__main__':
    for klass in (Commuter1, Commuter2, Commuter3, Commuter4, Commuter5, Commuter6):
        print('-' * 50)
        x = klass(88)
        y = klass(99)
        print(x + 1)
        print(1 + y)
        print(x + y)

$ python3 commuter.py
--------------------------------------------------
add 88 1
89
radd 99 1
100
add 88 <__main__.Commuter1 object at 0x101edc2f0>
radd 99 88
187
--------------------------------------------------
…etc…
--------------------------------------------------
Commuter6(89)
Commuter6(100)
Commuter6(187)

In-Place Addition

To also implement += in-place augmented addition, code either an __iadd__ or an __add__. The latter is used if the former is absent. In fact, the prior section’s Commuter classes already support += for this reason—Python runs __add__ and assigns the result manually. The __iadd__ method, though, allows for more efficient in-place changes to be coded where applicable; its return value is assigned to the target on the left of the +=:

>>> class Number:
        def __init__(self, val):
            self.val = val
        def __iadd__(self, other):             # __iadd__ explicit: x += y
            self.val += other                  # Usually returns self
            return self                        # Else None is returned+assigned

>>> x = Number(5)
>>> x += 1
>>> x += 1
>>> x.val
7

For mutable objects, this method can often specialize for quicker in-place changes:

>>> y = Number([1])                            # In-place change faster than +
>>> y += [2]
>>> y += [3]
>>> y.val
[1, 2, 3]

The normal __add__ method is run as a fallback, but may not be able to optimize in-place cases:

>>> class Number:
        def __init__(self, val):
            self.val = val
        def __add__(self, other):              # __add__ fallback: x = (x + y)
            return Number(self.val + other)    # Propagates class type

>>> x = Number(5)
>>> x += 1
>>> x += 1                                     # And += does concatenation here
>>> x.val
7

Though we’ve focused on + here, keep in mind that every binary operator has similar right-side and in-place overloading methods that work the same (e.g., __mul__, __rmul__, and __imul__). Still, these methods tend to be uncommon in practice; you only code right-side methods for either-side roles and in-place methods for code economy or speed—and only if you need to support such operators at all. For instance, a Vector class may use these tools, but a Button class probably would not. Button presses, though, may run the next section’s method.

Function Interfaces and Callback-Based Code

As an example, Python’s tkinter GUI module lets you to register functions as event handlers (a.k.a. callbacks)—when events occur, tkinter calls the registered objects. If you want an event handler to retain state between events, you can register a class instance that conforms to the expected interface with __call__. Chapter 17’s closure functions can achieve similar effects but don’t provide as much support for multiple operations or customization.

To demo the concept, here’s a hypothetical example of __call__ applied to the GUI domain. The following class defines an object that supports a function-call interface but also has state information that remembers the color a button should change to when it is later pressed:

class Callback:
    def __init__(self, color):               # Function + state information
        self.color = color
    def __call__(self):                      # Support calls with no arguments
        print('turn', self.color)

In the context of a GUI, we can register instances of this class as event handlers for buttons, even though the GUI expects to be able to invoke event handlers as simple functions with no arguments:

cb1 = Callback('blue')                       # Remember blue
cb2 = Callback('green')                      # Remember green

B1 = Button(command=cb1)                     # Register handlers
B2 = Button(command=cb2)

When the button is later pressed, the instance object is called as a simple function with no arguments, exactly like in the following calls. Because it retains state as instance attributes, though, it remembers what to do—it becomes a stateful function object:

cb1()                                        # On event: prints 'turn blue'
cb2()                                        # On event: prints 'turn green'

In fact, some consider such classes to be the best way to retain state information in the Python language. With OOP, the state remembered is made explicit with attribute assignments. This is different than other state retention techniques (e.g., global variables, enclosing function scope references, and default mutable arguments), which rely on more limited or implicit behavior. Moreover, the added structure and customization in classes goes beyond state retention.

On the other hand, tools such as closure functions are useful in basic state retention roles too, and the nonlocal statement makes enclosing scopes a viable alternative in more programs. We’ll revisit such trade-offs when we start coding substantial decorators in the online-only “Decorators” chapter, but here’s a quick closure equivalent:

def callback(color):                         # Enclosing scope versus attrs
    def oncall():
        print('turn', color)
    return oncall

cb3 = callback('yellow')                     # Handler to be registered
cb3()                                        # On event: prints 'turn yellow'

Before we move on, there are two other ways that Python programmers sometimes tie information to a callback function like this. One option is to use default arguments in lambda functions:

cb4 = (lambda color='red':                   # Defaults retain state too
           print('turn', color))             # lambda defers code till call
cb4()                                        # On event: prints 'turn red'

The other is to use bound methods of a class—covered in the next chapter but simple enough to preview here. A bound-method object is created for instance methods reference but not called and remembers both the instance and the referenced function. This object may therefore be called later as a simple function without an instance:

class Callback:
    def __init__(self, color):               # Class with state information
        self.color = color
    def changeColor(self):                   # A normally named method
        print('turn', self.color)

cb1 = Callback('blue')
cb2 = Callback('yellow')

B1 = Button(command=cb1.changeColor)         # Bound method: reference, not call
B2 = Button(command=cb2.changeColor)         # Remembers instance + function pair

In this case, when this button is later pressed it’s as if the GUI does the following, which invokes the instance’s changeColor method to process the cb1 object’s state information instead of calling the instance itself:

cb1 = Callback('blue')
cmd = cb1.changeColor                        # Registered event handler
cmd()                                        # On event: prints 'turn blue'

Note that a lambda is not required here unless extra arguments must be passed, because a bound method reference by itself already defers a call until later. This technique doesn’t require overloading calls with __call__, but either scheme may be preferred for a given program. Again, watch for more about bound methods in the next chapter.

You’ll also see another __call__ example in Chapter 32, where we will use it to implement a function decorator—a callable object often used to add a layer of logic on top of an embedded function. Because __call__ allows us to attach state information to a callable object, it’s a natural implementation technique for a function that must remember to call another function when called itself. For more __call__ examples, also watch for the more advanced decorators and metaclasses of the online-only “Decorators” chapter and Chapter 40.

Destructor Usage Notes

The destructor method works as documented, but it has some well-known caveats and a few outright dark corners that make it somewhat rare to see in Python code:

• Need: For one thing, destructors may not be as useful in Python as they are in some other OOP languages. Because Python automatically reclaims all memory space held by an instance when the instance is reclaimed, destructors are not necessary for space management. In the current CPython implementation of Python, you also don’t need to close file objects held by the instance in destructors because they are automatically closed when reclaimed. As mentioned in Chapter 9, though, it’s still sometimes best to run file close methods anyhow because this autoclose behavior may vary in alternative Python implementations.

• Predictability: For another, you cannot always easily predict when an instance will be reclaimed. In some cases, there may be lingering references to your objects in system tables that prevent destructors from running when your program expects them to be triggered. Python also does not guarantee that destructor methods will be called for objects that still exist when the interpreter exits.

• Exceptions: In fact, __del__ can be tricky to use for even more subtle reasons. Exceptions raised within it, for example, simply print a warning message to sys.stderr (the standard error stream) rather than triggering an exception event, because of the unpredictable context under which it is run by the garbage collector—it’s not always possible to know where such an exception should be delivered.

• Cycles: In addition, cyclic (a.k.a. circular) references among objects may prevent garbage collection from happening when you expect it to. An optional cycle detector, enabled by default, can automatically collect such objects eventually (including objects with __del__ methods, as of Python 3.4). Since this is relatively obscure, we’ll ignore further details here; see Python’s standard manuals’ coverage of both __del__ and the gc garbage collector module for more information.

Because of these downsides, it’s often better to code termination activities in an explicitly called method (e.g., shutdown). As described in the next part of the book, the try/finally statement also supports termination actions, as does the with statement for objects that support its context-manager model.