What’s New in This Chapter

I added a new epigraph and a few words in the second paragraph of the chapter to address the concept of “Pythonic”—which was only discussed at the very end in the first edition.

“Formatted Displays” was updated to mention f-strings, introduced in Python 3.6. It’s a small change because f-strings support the same formatting mini-language as the format() built-in and the str.format() method, so any previously implemented __format__ methods simply work with f-strings.

The rest of the chapter barely changed—the special methods are mostly the same since Python 3.0, and the core ideas appeared in Python 2.2.

Let’s get started with the object representation methods.

Object Representations

Every object-oriented language has at least one standard way of getting a string representation from any object. Python has two:

repr()

Return a string representing the object as the developer wants to see it. It’s what you get when the Python console or a debugger shows an object.

str()

Return a string representing the object as the user wants to see it. It’s what you get when you print() an object.

The special methods __repr__ and __str__ support repr() and str(), as we saw in Chapter 1.

There are two additional special methods to support alternative representations of objects: __bytes__ and __format__. The __bytes__ method is analogous to __str__: it’s called by bytes() to get the object represented as a byte sequence. Regarding __format__, it is used by f-strings, by the built-in function format(), and by the str.format() method. They call obj.__format__(format_spec) to get string displays of objects using special formatting codes. We’ll cover __bytes__ in the next example, and __format__ after that.

In order to demonstrate the many methods used to generate object representations, we’ll use a Vector2d class similar to the one we saw in Chapter 1. We will build on it in this and future sections. Example 11-1 illustrates the basic behavior we expect from a Vector2d instance.

    >>> v1 = Vector2d(3, 4)
    >>> print(v1.x, v1.y)  
    3.0 4.0
    >>> x, y = v1  
    >>> x, y
    (3.0, 4.0)
    >>> v1  
    Vector2d(3.0, 4.0)
    >>> v1_clone = eval(repr(v1))  
    >>> v1 == v1_clone  
    True
    >>> print(v1)  
    (3.0, 4.0)
    >>> octets = bytes(v1)  
    >>> octets
    b'd\\x00\\x00\\x00\\x00\\x00\\x00\\x08@\\x00\\x00\\x00\\x00\\x00\\x00\\x10@'
    >>> abs(v1)  
    5.0
    >>> bool(v1), bool(Vector2d(0, 0))  
    (True, False)

The components of a Vector2d can be accessed directly as attributes (no getter method calls).

A Vector2d can be unpacked to a tuple of variables.

The repr of a Vector2d emulates the source code for constructing the instance.

Using eval here shows that the repr of a Vector2d is a faithful representation of its constructor call.²

Vector2d supports comparison with ==; this is useful for testing.

print calls str, which for Vector2d produces an ordered pair display.

bytes uses the __bytes__ method to produce a binary representation.

abs uses the __abs__ method to return the magnitude of the Vector2d.

bool uses the __bool__ method to return False for a Vector2d of zero magnitude or True otherwise.

Vector2d from Example 11-1 is implemented in vector2d_v0.py (Example 11-2). The code is based on Example 1-2, except for the methods for the + and * operations, which we’ll see later in Chapter 16. We’ll add the method for == since it’s useful for testing. At this point, Vector2d uses several special methods to provide operations that a Pythonista expects in a well-designed object.

from array import array
import math


class Vector2d:
    typecode = 'd'  

    def __init__(self, x, y):
        self.x = float(x)    
        self.y = float(y)

    def __iter__(self):
        return (i for i in (self.x, self.y))  

    def __repr__(self):
        class_name = type(self).__name__
        return '{}({!r}, {!r})'.format(class_name, *self)  

    def __str__(self):
        return str(tuple(self))  

    def __bytes__(self):
        return (bytes([ord(self.typecode)]) +  
                bytes(array(self.typecode, self)))  

    def __eq__(self, other):
        return tuple(self) == tuple(other)  

    def __abs__(self):
        return math.hypot(self.x, self.y)  

    def __bool__(self):
        return bool(abs(self))  

typecode is a class attribute we’ll use when converting Vector2d instances to/from bytes.

Converting x and y to float in __init__ catches errors early, which is helpful in case Vector2d is called with unsuitable arguments.

__iter__ makes a Vector2d iterable; this is what makes unpacking work (e.g, x, y = my_vector). We implement it simply by using a generator expression to yield the components one after the other.³

__repr__ builds a string by interpolating the components with {!r} to get their repr; because Vector2d is iterable, *self feeds the x and y components to format.

From an iterable Vector2d, it’s easy to build a tuple for display as an ordered pair.

To generate bytes, we convert the typecode to bytes and concatenate…

…bytes converted from an array built by iterating over the instance.

To quickly compare all components, build tuples out of the operands. This works for operands that are instances of Vector2d, but has issues. See the following warning.

The magnitude is the length of the hypotenuse of the right triangle formed by the x and y components.

__bool__ uses abs(self) to compute the magnitude, then converts it to bool, so 0.0 becomes False, nonzero is True.

We have a fairly complete set of basic methods, but we still need a way to rebuild a Vector2d from the binary representation produced by bytes().

Since we can export a Vector2d as bytes, naturally we need a method that imports a Vector2d from a binary sequence. Looking at the standard library for inspiration, we find that array.array has a class method named .frombytes that suits our purpose—we saw it in “Arrays”. We adopt its name and use its functionality in a class method for Vector2d in vector2d_v1.py (Example 11-3).

    @classmethod  
    def frombytes(cls, octets):  
        typecode = chr(octets[0])  
        memv = memoryview(octets[1:]).cast(typecode)  
        return cls(*memv)  

The classmethod decorator modifies a method so it can be called directly on a class.

No self argument; instead, the class itself is passed as the first argument—conventionally named cls.

Read the typecode from the first byte.

Create a memoryview from the octets binary sequence and use the typecode to cast it.⁴

Unpack the memoryview resulting from the cast into the pair of arguments needed for the constructor.

I just used a classmethod decorator and it is very Python specific, so let’s have a word about it.

classmethod Versus staticmethod

The classmethod decorator is not mentioned in the Python tutorial, and neither is staticmethod. Anyone who has learned OO in Java may wonder why Python has both of these decorators and not just one of them.

Let’s start with classmethod. Example 11-3 shows its use: to define a method that operates on the class and not on instances. classmethod changes the way the method is called, so it receives the class itself as the first argument, instead of an instance. Its most common use is for alternative constructors, like frombytes in Example 11-3. Note how the last line of frombytes actually uses the cls argument by invoking it to build a new instance: cls(*memv).

In contrast, the staticmethod decorator changes a method so that it receives no special first argument. In essence, a static method is just like a plain function that happens to live in a class body, instead of being defined at the module level. Example 11-4 contrasts the operation of classmethod and staticmethod.

>>> class Demo:
...     @classmethod
...     def klassmeth(*args):
...         return args  
...     @staticmethod
...     def statmeth(*args):
...         return args  
...
>>> Demo.klassmeth()  
(<class '__main__.Demo'>,)
>>> Demo.klassmeth('spam')
(<class '__main__.Demo'>, 'spam')
>>> Demo.statmeth()   
()
>>> Demo.statmeth('spam')
('spam',)

klassmeth just returns all positional arguments.

statmeth does the same.

No matter how you invoke it, Demo.klassmeth receives the Demo class as the first argument.

Demo.statmeth behaves just like a plain old function.

Now that we’ve seen what classmethod is good for (and that staticmethod is not very useful), let’s go back to the issue of object representation and see how to support formatted output.

Formatted Displays

The f-strings, the format() built-in function, and the str.format() method delegate the actual formatting to each type by calling their .__format__(format_spec) method. The format_spec is a formatting specifier, which is either:

>>> brl = 1 / 4.82  # BRL to USD currency conversion rate
>>> brl
0.20746887966804978
>>> format(brl, '0.4f')  
'0.2075'
>>> '1 BRL = {rate:0.2f} USD'.format(rate=brl)  
'1 BRL = 0.21 USD'
>>> f'1 USD = {1 / brl:0.2f} BRL'  
'1 USD = 4.82 BRL'

Formatting specifier is '0.4f'.

Formatting specifier is '0.2f'. The rate part in the replacement field is not part of the formatting specifier. It determines which keyword argument of .format() goes into that replacement field.

Again, the specifier is '0.2f'. The 1 / brl expression is not part of it.

The second and third callouts make an important point: a format string such as '{0.mass:5.3e}' actually uses two separate notations. The '0.mass' to the left of the colon is the field_name part of the replacement field syntax, and it can be an arbitrary expression in an f-string. The '5.3e' after the colon is the formatting specifier. The notation used in the formatting specifier is called the Format Specification Mini-Language.

A few built-in types have their own presentation codes in the Format Specification Mini-Language. For example—among several other codes—the int type supports b and x for base 2 and base 16 output, respectively, while float implements f for a fixed-point display and % for a percentage display:

>>> format(42, 'b')
'101010'
>>> format(2 / 3, '.1%')
'66.7%'

The Format Specification Mini-Language is extensible because each class gets to interpret the format_spec argument as it likes. For instance, the classes in the datetime module use the same format codes in the strftime() functions and in their __format__ methods. Here are a couple of examples using the format() built-in and the str.format() method:

>>> from datetime import datetime
>>> now = datetime.now()
>>> format(now, '%H:%M:%S')
'18:49:05'
>>> "It's now {:%I:%M %p}".format(now)
"It's now 06:49 PM"

If a class has no __format__, the method inherited from object returns str(my_object). Because Vector2d has a __str__, this works:

>>> v1 = Vector2d(3, 4)
>>> format(v1)
'(3.0, 4.0)'

However, if you pass a format specifier, object.__format__ raises TypeError:

>>> format(v1, '.3f')
Traceback (most recent call last):
  ...
TypeError: non-empty format string passed to object.__format__

We will fix that by implementing our own format mini-language. The first step will be to assume the format specifier provided by the user is intended to format each float component of the vector. This is the result we want:

>>> v1 = Vector2d(3, 4)
>>> format(v1)
'(3.0, 4.0)'
>>> format(v1, '.2f')
'(3.00, 4.00)'
>>> format(v1, '.3e')
'(3.000e+00, 4.000e+00)'

Example 11-5 implements __format__ to produce the displays just shown.

    # inside the Vector2d class

    def __format__(self, fmt_spec=''):
        components = (format(c, fmt_spec) for c in self)  
        return '({}, {})'.format(*components)  

Use the format built-in to apply the fmt_spec to each vector component, building an iterable of formatted strings.

Plug the formatted strings in the formula '(x, y)'.

Now let’s add a custom formatting code to our mini-language: if the format specifier ends with a 'p', we’ll display the vector in polar coordinates: <r, θ>, where r is the magnitude and θ (theta) is the angle in radians. The rest of the format specifier (whatever comes before the 'p') will be used as before.

To generate polar coordinates, we already have the __abs__ method for the magnitude, and we’ll code a simple angle method using the math.atan2() function to get the angle. This is the code:

    # inside the Vector2d class

    def angle(self):
        return math.atan2(self.y, self.x)

With that, we can enhance our __format__ to produce polar coordinates. See Example 11-6.

    def __format__(self, fmt_spec=''):
        if fmt_spec.endswith('p'):  
            fmt_spec = fmt_spec[:-1]  
            coords = (abs(self), self.angle())  
            outer_fmt = '<{}, {}>'  
        else:
            coords = self  
            outer_fmt = '({}, {})'  
        components = (format(c, fmt_spec) for c in coords)  
        return outer_fmt.format(*components)  

Format ends with 'p': use polar coordinates.

Remove 'p' suffix from fmt_spec.

Build tuple of polar coordinates: (magnitude, angle).

Configure outer format with angle brackets.

Otherwise, use x, y components of self for rectangular coordinates.

Configure outer format with parentheses.

Generate iterable with components as formatted strings.

Plug formatted strings into outer format.

With Example 11-6, we get results similar to these:

>>> format(Vector2d(1, 1), 'p')
'<1.4142135623730951, 0.7853981633974483>'
>>> format(Vector2d(1, 1), '.3ep')
'<1.414e+00, 7.854e-01>'
>>> format(Vector2d(1, 1), '0.5fp')
'<1.41421, 0.78540>'

As this section shows, it’s not hard to extend the Format Specification Mini-Language to support user-defined types.

Now let’s move to a subject that’s not just about appearances: we will make our Vector2d hashable, so we can build sets of vectors, or use them as dict keys.

As defined, so far our Vector2d instances are unhashable, so we can’t put them in a set:

>>> v1 = Vector2d(3, 4)
>>> hash(v1)
Traceback (most recent call last):
  ...
TypeError: unhashable type: 'Vector2d'
>>> set([v1])
Traceback (most recent call last):
  ...
TypeError: unhashable type: 'Vector2d'

To make a Vector2d hashable, we must implement __hash__ (__eq__ is also required, and we already have it). We also need to make vector instances immutable, as we’ve seen in “What Is Hashable”.

Right now, anyone can do v1.x = 7, and there is nothing in the code to suggest that changing a Vector2d is forbidden. This is the behavior we want:

>>> v1.x, v1.y
(3.0, 4.0)
>>> v1.x = 7
Traceback (most recent call last):
  ...
AttributeError: can't set attribute

We’ll do that by making the x and y components read-only properties in Example 11-7.

class Vector2d:
    typecode = 'd'

    def __init__(self, x, y):
        self.__x = float(x)  
        self.__y = float(y)

    @property  
    def x(self):  
        return self.__x  

    @property  
    def y(self):
        return self.__y

    def __iter__(self):
        return (i for i in (self.x, self.y))  

    # remaining methods: same as previous Vector2d

Use exactly two leading underscores (with zero or one trailing underscore) to make an attribute private.⁶

The @property decorator marks the getter method of a property.

The getter method is named after the public property it exposes: x.

Just return self.__x.

Repeat the same formula for y property.

Every method that just reads the x, y components can stay as it was, reading the public properties via self.x and self.y instead of the private attribute, so this listing omits the rest of the code for the class.

Now that our vectors are reasonably safe from accidental mutation, we can implement the __hash__ method. It should return an int and ideally take into account the hashes of the object attributes that are also used in the __eq__ method, because objects that compare equal should have the same hash. The __hash__ special method documentation suggests computing the hash of a tuple with the components, so that’s what we do in Example 11-8.

    # inside class Vector2d:

    def __hash__(self):
        return hash((self.x, self.y))

With the addition of the __hash__ method, we now have hashable vectors:

>>> v1 = Vector2d(3, 4)
>>> v2 = Vector2d(3.1, 4.2)
>>> hash(v1), hash(v2)
(1079245023883434373, 1994163070182233067)
>>> {v1, v2}
{Vector2d(3.1, 4.2), Vector2d(3.0, 4.0)}

If you are creating a type that has a sensible scalar numeric value, you may also implement the __int__ and __float__ methods, invoked by the int() and float() constructors, which are used for type coercion in some contexts. There is also a __complex__ method to support the complex() built-in constructor. Perhaps Vector2d should provide __complex__, but I’ll leave that as an exercise for you.

So far, Vector2d instances are compatible with keyword class patterns—covered in “Keyword Class Patterns”.

In Example 11-9, all of these keyword patterns work as expected.

def keyword_pattern_demo(v: Vector2d) -> None:
    match v:
        case Vector2d(x=0, y=0):
            print(f'{v!r} is null')
        case Vector2d(x=0):
            print(f'{v!r} is vertical')
        case Vector2d(y=0):
            print(f'{v!r} is horizontal')
        case Vector2d(x=x, y=y) if x==y:
            print(f'{v!r} is diagonal')
        case _:
            print(f'{v!r} is awesome')

However, if you try to use a positional pattern like this:

        case Vector2d(_, 0):
            print(f'{v!r} is horizontal')

you get:

TypeError: Vector2d() accepts 0 positional sub-patterns (1 given)

To make Vector2d work with positional patterns, we need to add a class attribute named __match_args__ , listing the instance attributes in the order they will be used for positional pattern matching:

class Vector2d:
    __match_args__ = ('x', 'y')

    # etc...

Now we can save a few keystrokes when writing patterns to match Vector2d subjects, as you can see in Example 11-10.

def positional_pattern_demo(v: Vector2d) -> None:
    match v:
        case Vector2d(0, 0):
            print(f'{v!r} is null')
        case Vector2d(0):
            print(f'{v!r} is vertical')
        case Vector2d(_, 0):
            print(f'{v!r} is horizontal')
        case Vector2d(x, y) if x==y:
            print(f'{v!r} is diagonal')
        case _:
            print(f'{v!r} is awesome')

The __match_args__ class attribute does not need to include all public instance attributes. In particular, if the class __init__ has required and optional arguments that are assigned to instance attributes, it may be reasonable to name the required arguments in __match_args__, but not the optional ones.

Let’s step back and review what we’ve coded so far in Vector2d.

Complete Listing of Vector2d, Version 3

We have been working on Vector2d for a while, showing just snippets, so Example 11-11 is a consolidated, full listing of vector2d_v3.py, including the doctests I used when developing it.

"""
A two-dimensional vector class

    >>> v1 = Vector2d(3, 4)
    >>> print(v1.x, v1.y)
    3.0 4.0
    >>> x, y = v1
    >>> x, y
    (3.0, 4.0)
    >>> v1
    Vector2d(3.0, 4.0)
    >>> v1_clone = eval(repr(v1))
    >>> v1 == v1_clone
    True
    >>> print(v1)
    (3.0, 4.0)
    >>> octets = bytes(v1)
    >>> octets
    b'd\\x00\\x00\\x00\\x00\\x00\\x00\\x08@\\x00\\x00\\x00\\x00\\x00\\x00\\x10@'
    >>> abs(v1)
    5.0
    >>> bool(v1), bool(Vector2d(0, 0))
    (True, False)


Test of ``.frombytes()`` class method:

    >>> v1_clone = Vector2d.frombytes(bytes(v1))
    >>> v1_clone
    Vector2d(3.0, 4.0)
    >>> v1 == v1_clone
    True


Tests of ``format()`` with Cartesian coordinates:

    >>> format(v1)
    '(3.0, 4.0)'
    >>> format(v1, '.2f')
    '(3.00, 4.00)'
    >>> format(v1, '.3e')
    '(3.000e+00, 4.000e+00)'


Tests of the ``angle`` method::

    >>> Vector2d(0, 0).angle()
    0.0
    >>> Vector2d(1, 0).angle()
    0.0
    >>> epsilon = 10**-8
    >>> abs(Vector2d(0, 1).angle() - math.pi/2) < epsilon
    True
    >>> abs(Vector2d(1, 1).angle() - math.pi/4) < epsilon
    True


Tests of ``format()`` with polar coordinates:

    >>> format(Vector2d(1, 1), 'p')  # doctest:+ELLIPSIS
    '<1.414213..., 0.785398...>'
    >>> format(Vector2d(1, 1), '.3ep')
    '<1.414e+00, 7.854e-01>'
    >>> format(Vector2d(1, 1), '0.5fp')
    '<1.41421, 0.78540>'


Tests of `x` and `y` read-only properties:

    >>> v1.x, v1.y
    (3.0, 4.0)
    >>> v1.x = 123
    Traceback (most recent call last):
      ...
    AttributeError: can't set attribute 'x'


Tests of hashing:

    >>> v1 = Vector2d(3, 4)
    >>> v2 = Vector2d(3.1, 4.2)
    >>> len({v1, v2})
    2

"""

from array import array
import math

class Vector2d:
    __match_args__ = ('x', 'y')

    typecode = 'd'

    def __init__(self, x, y):
        self.__x = float(x)
        self.__y = float(y)

    @property
    def x(self):
        return self.__x

    @property
    def y(self):
        return self.__y

    def __iter__(self):
        return (i for i in (self.x, self.y))

    def __repr__(self):
        class_name = type(self).__name__
        return '{}({!r}, {!r})'.format(class_name, *self)

    def __str__(self):
        return str(tuple(self))

    def __bytes__(self):
        return (bytes([ord(self.typecode)]) +
                bytes(array(self.typecode, self)))

    def __eq__(self, other):
        return tuple(self) == tuple(other)

    def __hash__(self):
        return hash((self.x, self.y))

    def __abs__(self):
        return math.hypot(self.x, self.y)

    def __bool__(self):
        return bool(abs(self))

    def angle(self):
        return math.atan2(self.y, self.x)

    def __format__(self, fmt_spec=''):
        if fmt_spec.endswith('p'):
            fmt_spec = fmt_spec[:-1]
            coords = (abs(self), self.angle())
            outer_fmt = '<{}, {}>'
        else:
            coords = self
            outer_fmt = '({}, {})'
        components = (format(c, fmt_spec) for c in coords)
        return outer_fmt.format(*components)

    @classmethod
    def frombytes(cls, octets):
        typecode = chr(octets[0])
        memv = memoryview(octets[1:]).cast(typecode)
        return cls(*memv)

To recap, in this and the previous sections, we saw some essential special methods that you may want to implement to have a full-fledged object.

As coded in Example 11-11, Vector2d is a didactic example with a laundry list of special methods related to object representation, not a template for every user-defined class.

In the next section, we’ll take a break from Vector2d to discuss the design and drawbacks of the private attribute mechanism in Python—the double-underscore prefix in self.__x.

In Python, there is no way to create private variables like there is with the private modifier in Java. What we have in Python is a simple mechanism to prevent accidental overwriting of a “private” attribute in a subclass.

To prevent this, if you name an instance attribute in the form __mood (two leading underscores and zero or at most one trailing underscore), Python stores the name in the instance __dict__ prefixed with a leading underscore and the class name, so in the Dog class, __mood becomes _Dog__mood, and in Beagle it’s _Beagle__mood. This language feature goes by the lovely name of name mangling.

Example 11-12 shows the result in the Vector2d class from Example 11-7.

>>> v1 = Vector2d(3, 4)
>>> v1.__dict__
{'_Vector2d__y': 4.0, '_Vector2d__x': 3.0}
>>> v1._Vector2d__x
3.0

Name mangling is about safety, not security: it’s designed to prevent accidental access and not malicious prying. Figure 11-1 illustrates another safety device.

Anyone who knows how private names are mangled can read the private attribute directly, as the last line of Example 11-12 shows—that’s actually useful for debugging and serialization. They can also directly assign a value to a private component of a Vector2d by writing v1._Vector2d__x = 7. But if you are doing that in production code, you can’t complain if something blows up.

The name mangling functionality is not loved by all Pythonistas, and neither is the skewed look of names written as self.__x. Some prefer to avoid this syntax and use just one underscore prefix to “protect” attributes by convention (e.g., self._x). Critics of the automatic double-underscore mangling suggest that concerns about accidental attribute clobbering should be addressed by naming conventions. Ian Bicking—creator of pip, virtualenv, and other projects—wrote:

Never, ever use two leading underscores. This is annoyingly private. If name clashes are a concern, use explicit name mangling instead (e.g., _MyThing_blahblah). This is essentially the same thing as double-underscore, only it’s transparent where double underscore obscures.⁷

Figure 11-1. A cover on a switch is a safety device, not a security one: it prevents accidents, not sabotage.

The single underscore prefix has no special meaning to the Python interpreter when used in attribute names, but it’s a very strong convention among Python programmers that you should not access such attributes from outside the class.⁸ It’s easy to respect the privacy of an object that marks its attributes with a single _, just as it’s easy respect the convention that variables in ALL_CAPS should be treated as constants.

Attributes with a single _ prefix are called “protected” in some corners of the Python documentation.⁹ The practice of “protecting” attributes by convention with the form self._x is widespread, but calling that a “protected” attribute is not so common. Some even call that a “private” attribute.

To conclude: the Vector2d components are “private” and our Vector2d instances are “immutable”—with scare quotes—because there is no way to make them really private and immutable.¹⁰

We’ll now come back to our Vector2d class. In the next section, we cover a special attribute (not a method) that affects the internal storage of an object, with potentially huge impact on the use of memory but little effect on its public interface: __slots__.

By default, Python stores the attributes of each instance in a dict named __dict__. As we saw in “Practical Consequences of How dict Works”, a dict has a significant memory overhead—even with the optimizations mentioned in that section. But if you define a class attribute named __slots__ holding a sequence of attribute names, Python uses an alternative storage model for the instance attributes: the attributes named in __slots__ are stored in a hidden array or references that use less memory than a dict. Let’s see how that works through simple examples, starting with Example 11-13.

>>> class Pixel:
...     __slots__ = ('x', 'y')  
...
>>> p = Pixel()  
>>> p.__dict__  
Traceback (most recent call last):
  ...
AttributeError: 'Pixel' object has no attribute '__dict__'
>>> p.x = 10  
>>> p.y = 20
>>> p.color = 'red'  
Traceback (most recent call last):
  ...
AttributeError: 'Pixel' object has no attribute 'color'

__slots__ must be present when the class is created; adding or changing it later has no effect. The attribute names may be in a tuple or list, but I prefer a tuple to make it clear there’s no point in changing it.

Create an instance of Pixel, because we see the effects of __slots__ on the instances.

First effect: instances of Pixel have no __dict__.

Set the p.x and p.y attributes normally.

Second effect: trying to set an attribute not listed in __slots__ raises AttributeError.

So far, so good. Now let’s create a subclass of Pixel in Example 11-14 to see the counterintuitive side of __slots__.

>>> class OpenPixel(Pixel):  
...     pass
...
>>> op = OpenPixel()
>>> op.__dict__  
{}
>>> op.x = 8  
>>> op.__dict__  
{}
>>> op.x  
8
>>> op.color = 'green'  
>>> op.__dict__  
{'color': 'green'}

OpenPixel declares no attributes of its own.

Surprise: instances of OpenPixel have a __dict__.

If you set attribute x (named in the __slots__ of the base class Pixel)…

…it is not stored in the instance __dict__…

…but it is stored in the hidden array of references in the instance.

If you set an attribute not named in the __slots__…

…it is stored in the instance __dict__.

Example 11-14 shows that the effect of __slots__ is only partially inherited by a subclass. To make sure that instances of a subclass have no __dict__, you must declare __slots__ again in the subclass.

If you declare __slots__ = () (an empty tuple), then the instances of the subclass will have no __dict__ and will only accept the attributes named in the __slots__ of the base class.

If you want a subclass to have additional attributes, name them in __slots__, as shown in Example 11-15.

>>> class ColorPixel(Pixel):
...    __slots__ = ('color',)  
>>> cp = ColorPixel()
>>> cp.__dict__  
Traceback (most recent call last):
  ...
AttributeError: 'ColorPixel' object has no attribute '__dict__'
>>> cp.x = 2
>>> cp.color = 'blue'  
>>> cp.flavor = 'banana'
Traceback (most recent call last):
  ...
AttributeError: 'ColorPixel' object has no attribute 'flavor'

Essentially, __slots__ of the superclasses are added to the __slots__ of the current class. Don’t forget that single-item tuples must have a trailing comma.

ColorPixel instances have no __dict__.

You can set the attributes declared in the __slots__ of this class and superclasses, but no other.

It’s possible to “save memory and eat it too”: if you add the '__dict__' name to the __slots__ list, your instances will keep attributes named in __slots__ in the per-instance array of references, but will also support dynamically created attributes, which will be stored in the usual __dict__. This is necessary if you want to use the @cached_property decorator (covered in “Step 5: Caching Properties with functools”).

Of course, having '__dict__' in __slots__ may entirely defeat its purpose, depending on the number of static and dynamic attributes in each instance and how they are used. Careless optimization is worse than premature optimization: you add complexity but may not get any benefit.

Another special per-instance attribute that you may want to keep is __weakref__, necessary for an object to support weak references (mentioned briefly in “del and Garbage Collection”). That attribute exists by default in instances of user-defined classes. However, if the class defines __slots__, and you need the instances to be targets of weak references, then you need to include '__weakref__' among the attributes named in __slots__.

Now let’s see the effect of adding __slots__ to Vector2d.

class Vector2d:
    __match_args__ = ('x', 'y')  
    __slots__ = ('__x', '__y')  

    typecode = 'd'
    # methods are the same as previous version

$ time python3 mem_test.py vector2d_v3
Selected Vector2d type: vector2d_v3.Vector2d
Creating 10,000,000 Vector2d instances
Initial RAM usage:      6,983,680
  Final RAM usage:  1,666,535,424

real	0m11.990s
user	0m10.861s
sys	0m0.978s
$ time python3 mem_test.py vector2d_v3_slots
Selected Vector2d type: vector2d_v3_slots.Vector2d
Creating 10,000,000 Vector2d instances
Initial RAM usage:      6,995,968
  Final RAM usage:    577,839,104

real	0m8.381s
user	0m8.006s
sys	0m0.352s

A distinctive feature of Python is how class attributes can be used as default values for instance attributes. In Vector2d there is the typecode class attribute. It’s used twice in the __bytes__ method, but we read it as self.typecode by design. Because Vector2d instances are created without a typecode attribute of their own, self.typecode will get the Vector2d.typecode class attribute by default.

The default Vector2d.typecode is 'd', meaning each vector component will be represented as an 8-byte double precision float when exporting to bytes. If we set the typecode of a Vector2d instance to 'f' prior to exporting, each component will be exported as a 4-byte single precision float. Example 11-18 demonstrates.

>>> from vector2d_v3 import Vector2d
>>> v1 = Vector2d(1.1, 2.2)
>>> dumpd = bytes(v1)
>>> dumpd
b'd\x9a\x99\x99\x99\x99\x99\xf1?\x9a\x99\x99\x99\x99\x99\x01@'
>>> len(dumpd)  
17
>>> v1.typecode = 'f'  
>>> dumpf = bytes(v1)
>>> dumpf
b'f\xcd\xcc\x8c?\xcd\xcc\x0c@'
>>> len(dumpf)  
9
>>> Vector2d.typecode  
'd'

Default bytes representation is 17 bytes long.

Set typecode to 'f' in the v1 instance.

Now the bytes dump is 9 bytes long.

Vector2d.typecode is unchanged; only the v1 instance uses typecode 'f'.

Now it should be clear why the bytes export of a Vector2d is prefixed by the typecode: we wanted to support different export formats.

If you want to change a class attribute, you must set it on the class directly, not through an instance. You could change the default typecode for all instances (that don’t have their own typecode) by doing this:

>>> Vector2d.typecode = 'f'

However, there is an idiomatic Python way of achieving a more permanent effect, and being more explicit about the change. Because class attributes are public, they are inherited by subclasses, so it’s common practice to subclass just to customize a class data attribute. The Django class-based views use this technique extensively. Example 11-19 shows how.

>>> from vector2d_v3 import Vector2d
>>> class ShortVector2d(Vector2d):  
...     typecode = 'f'
...
>>> sv = ShortVector2d(1/11, 1/27)  
>>> sv
ShortVector2d(0.09090909090909091, 0.037037037037037035)  
>>> len(bytes(sv))  
9

Create ShortVector2d as a Vector2d subclass just to overwrite the typecode class attribute.

Build ShortVector2d instance sv for demonstration.

Inspect the repr of sv.

Check that the length of the exported bytes is 9, not 17 as before.

This example also explains why I did not hardcode the class_name in Vector2d.​__repr__, but instead got it from type(self).__name__, like this:

    # inside class Vector2d:

    def __repr__(self):
        class_name = type(self).__name__
        return '{}({!r}, {!r})'.format(class_name, *self)

If I had hardcoded the class_name, subclasses of Vector2d like ShortVector2d would have to overwrite __repr__ just to change the class_name. By reading the name from the type of the instance, I made __repr__ safer to inherit.

This ends our coverage of building a simple class that leverages the data model to play well with the rest of Python: offering different object representations, providing a custom formatting code, exposing read-only attributes, and supporting hash() to integrate with sets and mappings.

The aim of this chapter was to demonstrate the use of special methods and conventions in the construction of a well-behaved Pythonic class.

Is vector2d_v3.py (shown in Example 11-11) more Pythonic than vector2d_v0.py (shown in Example 11-2)? The Vector2d class in vector2d_v3.py certainly exhibits more Python features. But whether the first or the last Vector2d implementation is suitable depends on the context where it would be used. Tim Peter’s “Zen of Python” says:

Simple is better than complex.

An object should be as simple as the requirements dictate—and not a parade of language features. If the code is for an application, then it should focus on what is needed to support the end users, not more. If the code is for a library for other programmers to use, then it’s reasonable to implement special methods supporting behaviors that Pythonistas expect. For example, __eq__ may not be necessary to support a business requirement, but it makes the class easier to test.

My goal in expanding the Vector2d code was to provide context for discussing Python special methods and coding conventions. The examples in this chapter have demonstrated several of the special methods we first saw in Table 1-1 (Chapter 1):

• String/bytes representation methods: __repr__, __str__, __format__, and __bytes__

• Methods for reducing an object to a number: __abs__, __bool__, and __hash__

• The __eq__ operator, to support testing and hashing (along with __hash__)

While supporting conversion to bytes, we also implemented an alternative constructor, Vector2d.frombytes(), which provided the context for discussing the decorators @classmethod (very handy) and @staticmethod (not so useful, module-level functions are simpler). The frombytes method was inspired by its namesake in the array.array class.

We saw that the Format Specification Mini-Language is extensible by implementing a __format__ method that parses a format_spec provided to the format(obj, format_spec) built-in or within replacement fields '{:«format_spec»}' in f-strings or strings used with the str.format() method.

In preparation to make Vector2d instances hashable, we made an effort to make them immutable, at least preventing accidental changes by coding the x and y attributes as private, and exposing them as read-only properties. We then implemented __hash__ using the recommended technique of xor-ing the hashes of the instance attributes.

We then discussed the memory savings and the caveats of declaring a __slots__ attribute in Vector2d. Because using __slots__ has side effects, it really makes sense only when handling a very large number of instances—think millions of instances, not just thousands. In many such cases, using pandas may be the best option.

The last topic we covered was the overriding of a class attribute accessed via the instances (e.g., self.typecode). We did that first by creating an instance attribute, and then by subclassing and overwriting at the class level.

Throughout the chapter, I mentioned how design choices in the examples were informed by studying the API of standard Python objects. If this chapter can be summarized in one sentence, this is it:

To build Pythonic objects, observe how real Python objects behave.

Ancient Chinese proverb

Further Reading

This chapter covered several special methods of the data model, so naturally the primary references are the same as the ones provided in Chapter 1, which gave a high-level view of the same topic. For convenience, I’ll repeat those four earlier recommendations here, and add a few other ones:

The “Data Model” chapter of The Python Language Reference

Most of the methods we used in this chapter are documented in “3.3.1. Basic customization”.

Python in a Nutshell, 3rd ed., by Alex Martelli, Anna Ravenscroft, and Steve Holden

Covers the special methods in depth.

Python Cookbook, 3rd ed., by David Beazley and Brian K. Jones

Modern Python practices demonstrated through recipes. Chapter 8, “Classes and Objects,” in particular has several solutions related to discussions in this chapter.

Python Essential Reference, 4th ed., by David Beazley

Covers the data model in detail, even if only Python 2.6 and 3.0 are covered (in the fourth edition). The fundamental concepts are all the same and most of the Data Model APIs haven’t changed at all since Python 2.2, when built-in types and user-defined classes were unified.

In 2015—the year I finished the first edition of Fluent Python—Hynek Schlawack started the attrs package. From the attrs documentation:

attrs is the Python package that will bring back the joy of writing classes by relieving you from the drudgery of implementing object protocols (aka dunder methods).

I mentioned attrs as a more powerful alternative to @dataclass in “Further Reading”. The data class builders from Chapter 5 as well as attrs automatically equip your classes with several special methods. But knowing how to code those special methods yourself is still essential to understand what those packages do, to decide whether you really need them, and to override the methods they generate—when necessary.

In this chapter, we saw every special method related to object representation, except __index__ and __fspath__. We’ll discuss __index__ in Chapter 12, “A Slice-Aware __getitem__”. I will not cover __fspath__. To learn about it, see PEP 519—Adding a file system path protocol.

An early realization of the need for distinct string representations for objects appeared in Smalltalk. The 1996 article “How to Display an Object as a String: printString and displayString” by Bobby Woolf discusses the implementation of the printString and displayString methods in that language. From that article, I borrowed the pithy descriptions “the way the developer wants to see it” and “the way the user wants to see it” when defining repr() and str() in “Object Representations”.