What’s New in This Chapter

The Quantity descriptor example in “LineItem Take #4: Automatic Naming of Storage Attributes” was dramatically simplified thanks to the __set_name__ special method added to the descriptor protocol in Python 3.6.

I removed the property factory example formerly in “LineItem Take #4: Automatic Naming of Storage Attributes” because it became irrelevant: the point was to show an alternative way of solving the Quantity problem, but with the addition of __set_name__, the descriptor solution becomes much simpler.

The AutoStorage class that used to appear in “LineItem Take #5: A New Descriptor Type” is also gone because __set_name__ made it obsolete.

Descriptor Example: Attribute Validation

As we saw in “Coding a Property Factory”, a property factory is a way to avoid repetitive coding of getters and setters by applying functional programming patterns. A property factory is a higher-order function that creates a parameterized set of accessor functions and builds a custom property instance from them, with closures to hold settings like the storage_name. The object-oriented way of solving the same problem is a descriptor class.

We’ll continue the series of LineItem examples where we left off, in “Coding a Property Factory”, by refactoring the quantity property factory into a Quantity descriptor class. This will make it easier to use.

LineItem Take #3: A Simple Descriptor

As we said in the introduction, a class implementing a __get__, a __set__, or a __delete__ method is a descriptor. You use a descriptor by declaring instances of it as class attributes of another class.

We’ll create a Quantity descriptor, and the LineItem class will use two instances of Quantity: one for managing the weight attribute, the other for price. A diagram helps, so take a look at Figure 23-1.

Figure 23-1. UML class diagram for LineItem using a descriptor class named Quantity. Underlined attributes in UML are class attributes. Note that weight and price are instances of Quantity attached to the LineItem class, but LineItem instances also have their own weight and price attributes where those values are stored.

Note that the word weight appears twice in Figure 23-1, because there are really two distinct attributes named weight: one is a class attribute of LineItem, the other is an instance attribute that will exist in each LineItem object. This also applies to price.

Terms to understand descriptors

Implementing and using descriptors involves several components, and it is useful to be precise when naming those components. I will use the following terms and definitions as I describe the examples in this chapter. They will be easier to understand once you see the code, but I wanted to put the definitions up front so you can refer back to them when needed.

Descriptor class

A class implementing the descriptor protocol. That’s Quantity in Figure 23-1.

Managed class

The class where the descriptor instances are declared as class attributes. In Figure 23-1, LineItem is the managed class.

Descriptor instance

Each instance of a descriptor class, declared as a class attribute of the managed class. In Figure 23-1, each descriptor instance is represented by a composition arrow with an underlined name (the underline means class attribute in UML). The black diamonds touch the LineItem class, which contains the descriptor instances.

Managed instance

One instance of the managed class. In this example, LineItem instances are the managed instances (they are not shown in the class diagram).

Storage attribute

An attribute of the managed instance that holds the value of a managed attribute for that particular instance. In Figure 23-1, the LineItem instance attributes weight and price are the storage attributes. They are distinct from the descriptor instances, which are always class attributes.

Managed attribute

A public attribute in the managed class that is handled by a descriptor instance, with values stored in storage attributes. In other words, a descriptor instance and a storage attribute provide the infrastructure for a managed attribute.

It’s important to realize that Quantity instances are class attributes of LineItem. This crucial point is highlighted by the mills and gizmos in Figure 23-2.

Figure 23-2. UML class diagram annotated with MGN (Mills & Gizmos Notation): classes are mills that produce gizmos—the instances. The Quantity mill produces two gizmos with round heads, which are attached to the LineItem mill: weight and price. The LineItem mill produces rectangular gizmos that have their own weight and price attributes where those values are stored.

Enough doodling for now. Here is the code: Example 23-1 shows the Quantity descriptor class, and Example 23-2 lists a new LineItem class using two instances of Quantity.

Example 23-1. bulkfood_v3.py: Quantity descriptor does not accept negative values

class Quantity:  

    def __init__(self, storage_name):
        self.storage_name = storage_name  

    def __set__(self, instance, value):  
        if value > 0:
            instance.__dict__[self.storage_name] = value  
        else:
            msg = f'{self.storage_name} must be > 0'
            raise ValueError(msg)

    def __get__(self, instance, owner):  
        return instance.__dict__[self.storage_name]

Descriptor is a protocol-based feature; no subclassing is needed to implement one.

Each Quantity instance will have a storage_name attribute: that’s the name of the storage attribute to hold the value in the managed instances.

__set__ is called when there is an attempt to assign to the managed attribute. Here, self is the descriptor instance (i.e., LineItem.weight or LineItem.price), instance is the managed instance (a LineItem instance), and value is the value being assigned.

We must store the attribute value directly into __dict__; calling set​attr​(instance, self.storage_name) would trigger the __set__ method again, leading to infinite recursion.

We need to implement __get__ because the name of the managed attribute may not be the same as the storage_name. The owner argument will be explained shortly.

Implementing __get__ is necessary because a user could write something like this:

class House:
    rooms = Quantity('number_of_rooms')

In the House class, the managed attribute is rooms, but the storage attribute is number_of_rooms. Given a House instance named chaos_manor, reading and writing chaos_manor.rooms goes through the Quantity descriptor instance attached to rooms, but reading and writing chaos_manor.number_of_rooms bypasses the descriptor.

Note that __get__ receives three arguments: self, instance, and owner. The owner argument is a reference to the managed class (e.g., LineItem), and it’s useful if you want the descriptor to support retrieving a class attribute—perhaps to emulate Python’s default behavior of retrieving a class attribute when the name is not found in the instance.

If a managed attribute, such as weight, is retrieved via the class like Line​Item.weight, the descriptor __get__ method receives None as the value for the instance argument.

To support introspection and other metaprogramming tricks by the user, it’s a good practice to make __get__ return the descriptor instance when the managed attribute is accessed through the class. To do that, we’d code __get__ like this:

    def __get__(self, instance, owner):
        if instance is None:
            return self
        else:
            return instance.__dict__[self.storage_name]

Example 23-2 demonstrates the use of Quantity in LineItem.

Example 23-2. bulkfood_v3.py: Quantity descriptors manage attributes in LineItem

class LineItem:
    weight = Quantity('weight')  
    price = Quantity('price')  

    def __init__(self, description, weight, price):  
        self.description = description
        self.weight = weight
        self.price = price

    def subtotal(self):
        return self.weight * self.price

The first descriptor instance will manage the weight attribute.

The second descriptor instance will manage the price attribute.

The rest of the class body is as simple and clean as the original code in bulkfood_v1.py (Example 22-19).

The code in Example 23-2 works as intended, preventing the sale of truffles for $0:³

>>> truffle = LineItem('White truffle', 100, 0)
Traceback (most recent call last):
    ...
ValueError: value must be > 0

Warning

When coding descriptor __get__ and __set__ methods, keep in mind what the self and instance arguments mean: self is the descriptor instance, and instance is the managed instance. Descriptors managing instance attributes should store values in the managed instances. That’s why Python provides the instance argument to the descriptor methods.

It may be tempting, but wrong, to store the value of each managed attribute in the descriptor instance itself. In other words, in the __set__ method, instead of coding:

    instance.__dict__[self.storage_name] = value

the tempting, but bad, alternative would be:

    self.__dict__[self.storage_name] = value

To understand why this would be wrong, think about the meaning of the first two arguments to __set__: self and instance. Here, self is the descriptor instance, which is actually a class attribute of the managed class. You may have thousands of LineItem instances in memory at one time, but you’ll only have two instances of the descriptors: the class attributes LineItem.weight and LineItem.price. So anything you store in the descriptor instances themselves is actually part of a LineItem class attribute, and therefore is shared among all LineItem instances.

A drawback of Example 23-2 is the need to repeat the names of the attributes when the descriptors are instantiated in the managed class body. It would be nice if the LineItem class could be declared like this:

class LineItem:
    weight = Quantity()
    price = Quantity()

    # remaining methods as before

As it stands, Example 23-2 requires naming each Quantity explicitly, which is not only inconvenient but dangerous. If a programmer copying and pasting code forgets to edit both names and writes something like price = Quantity('weight'), the program will misbehave badly, clobbering the value of weight whenever the price is set.

The problem is that—as we saw in Chapter 6—the righthand side of an assignment is executed before the variable exists. The expression Quantity() is evaluated to create a descriptor instance, and there is no way the code in the Quantity class can guess the name of the variable to which the descriptor will be bound (e.g., weight or price).

Thankfully, the descriptor protocol now supports the aptly named __set_name__ special method. We’ll see how to use it next.

Note

Automatic naming of a descriptor storage attribute used to be a thorny issue. In the first edition of Fluent Python, I devoted several pages and lines of code in this chapter and the next to presenting different solutions, including the use of a class decorator, and then metaclasses in Chapter 24. This was greatly simplified in Python 3.6.

class Quantity:

    def __set_name__(self, owner, name):  
        self.storage_name = name          

    def __set__(self, instance, value):   
        if value > 0:
            instance.__dict__[self.storage_name] = value
        else:
            msg = f'{self.storage_name} must be > 0'
            raise ValueError(msg)

    # no __get__ needed  

class LineItem:
    weight = Quantity()  
    price = Quantity()

    def __init__(self, description, weight, price):
        self.description = description
        self.weight = weight
        self.price = price

    def subtotal(self):
        return self.weight * self.price

import model_v4c as model  


class LineItem:
    weight = model.Quantity()  
    price = model.Quantity()

    def __init__(self, description, weight, price):
        self.description = description
        self.weight = weight
        self.price = price

    def subtotal(self):
        return self.weight * self.price

import abc

class Validated(abc.ABC):

    def __set_name__(self, owner, name):
        self.storage_name = name

    def __set__(self, instance, value):
        value = self.validate(self.storage_name, value)  
        instance.__dict__[self.storage_name] = value  

    @abc.abstractmethod
    def validate(self, name, value):  
        """return validated value or raise ValueError"""

class Quantity(Validated):
    """a number greater than zero"""

    def validate(self, name, value):  
        if value <= 0:
            raise ValueError(f'{name} must be > 0')
        return value


class NonBlank(Validated):
    """a string with at least one non-space character"""

    def validate(self, name, value):
        value = value.strip()
        if not value:  
            raise ValueError(f'{name} cannot be blank')
        return value  

import model_v5 as model  

class LineItem:
    description = model.NonBlank()  
    weight = model.Quantity()
    price = model.Quantity()

    def __init__(self, description, weight, price):
        self.description = description
        self.weight = weight
        self.price = price

    def subtotal(self):
        return self.weight * self.price

Recall that there is an important asymmetry in the way Python handles attributes. Reading an attribute through an instance normally returns the attribute defined in the instance, but if there is no such attribute in the instance, a class attribute will be retrieved. On the other hand, assigning to an attribute in an instance normally creates the attribute in the instance, without affecting the class at all.

Observing the different descriptor categories requires a few classes, so we’ll use the code in Example 23-8 as our test bed for the following sections.

### auxiliary functions for display only ###

def cls_name(obj_or_cls):
    cls = type(obj_or_cls)
    if cls is type:
        cls = obj_or_cls
    return cls.__name__.split('.')[-1]

def display(obj):
    cls = type(obj)
    if cls is type:
        return f'<class {obj.__name__}>'
    elif cls in [type(None), int]:
        return repr(obj)
    else:
        return f'<{cls_name(obj)} object>'

def print_args(name, *args):
    pseudo_args = ', '.join(display(x) for x in args)
    print(f'-> {cls_name(args[0])}.__{name}__({pseudo_args})')


### essential classes for this example ###

class Overriding:  
    """a.k.a. data descriptor or enforced descriptor"""

    def __get__(self, instance, owner):
        print_args('get', self, instance, owner)  

    def __set__(self, instance, value):
        print_args('set', self, instance, value)


class OverridingNoGet:  
    """an overriding descriptor without ``__get__``"""

    def __set__(self, instance, value):
        print_args('set', self, instance, value)


class NonOverriding:  
    """a.k.a. non-data or shadowable descriptor"""

    def __get__(self, instance, owner):
        print_args('get', self, instance, owner)


class Managed:  
    over = Overriding()
    over_no_get = OverridingNoGet()
    non_over = NonOverriding()

    def spam(self):  
        print(f'-> Managed.spam({display(self)})')

An overriding descriptor class with __get__ and __set__.

The print_args function is called by every descriptor method in this example.

An overriding descriptor without a __get__ method.

No __set__ method here, so this is a nonoverriding descriptor.

The managed class, using one instance of each of the descriptor classes.

The spam method is here for comparison, because methods are also descriptors.

In the following sections, we will examine the behavior of attribute reads and writes on the Managed class, and one instance of it, going through each of the different descriptors defined.

>>> obj = Managed()  
>>> obj.over  
-> Overriding.__get__(<Overriding object>, <Managed object>, <class Managed>)
>>> Managed.over  
-> Overriding.__get__(<Overriding object>, None, <class Managed>)
>>> obj.over = 7  
-> Overriding.__set__(<Overriding object>, <Managed object>, 7)
>>> obj.over  
-> Overriding.__get__(<Overriding object>, <Managed object>, <class Managed>)
>>> obj.__dict__['over'] = 8  
>>> vars(obj)  
{'over': 8}
>>> obj.over  
-> Overriding.__get__(<Overriding object>, <Managed object>, <class Managed>)

>>> obj.over_no_get  
<__main__.OverridingNoGet object at 0x665bcc>
>>> Managed.over_no_get  
<__main__.OverridingNoGet object at 0x665bcc>
>>> obj.over_no_get = 7  
-> OverridingNoGet.__set__(<OverridingNoGet object>, <Managed object>, 7)
>>> obj.over_no_get  
<__main__.OverridingNoGet object at 0x665bcc>
>>> obj.__dict__['over_no_get'] = 9  
>>> obj.over_no_get  
9
>>> obj.over_no_get = 7  
-> OverridingNoGet.__set__(<OverridingNoGet object>, <Managed object>, 7)
>>> obj.over_no_get  
9

>>> obj = Managed()
>>> obj.non_over  
-> NonOverriding.__get__(<NonOverriding object>, <Managed object>, <class Managed>)
>>> obj.non_over = 7  
>>> obj.non_over  
7
>>> Managed.non_over  
-> NonOverriding.__get__(<NonOverriding object>, None, <class Managed>)
>>> del obj.non_over  
>>> obj.non_over  
-> NonOverriding.__get__(<NonOverriding object>, <Managed object>, <class Managed>)

>>> obj = Managed()  
>>> Managed.over = 1  
>>> Managed.over_no_get = 2
>>> Managed.non_over = 3
>>> obj.over, obj.over_no_get, obj.non_over  
(1, 2, 3)

A function within a class becomes a bound method when invoked on an instance because all user-defined functions have a __get__ method, therefore they operate as descriptors when attached to a class. Example 23-13 demonstrates reading the spam method from the Managed class introduced in Example 23-8.

>>> obj = Managed()
>>> obj.spam  
<bound method Managed.spam of <descriptorkinds.Managed object at 0x74c80c>>
>>> Managed.spam  
<function Managed.spam at 0x734734>
>>> obj.spam = 7  
>>> obj.spam
7

Reading from obj.spam retrieves a bound method object.

But reading from Managed.spam retrieves a function.

Assigning a value to obj.spam shadows the class attribute, rendering the spam method inaccessible from the obj instance.

Functions do not implement __set__, therefore they are nonoverriding descriptors, as the last line of Example 23-13 shows.

The other key takeaway from Example 23-13 is that obj.spam and Managed.spam retrieve different objects. As usual with descriptors, the __get__ of a function returns a reference to itself when the access happens through the managed class. But when the access goes through an instance, the __get__ of the function returns a bound method object: a callable that wraps the function and binds the managed instance (e.g., obj) to the first argument of the function (i.e., self), like the functools.partial function does (as seen in “Freezing Arguments with functools.partial”). For a deeper understanding of this mechanism, take a look at Example 23-14.

import collections


class Text(collections.UserString):

    def __repr__(self):
        return 'Text({!r})'.format(self.data)

    def reverse(self):
        return self[::-1]

Now let’s investigate the Text.reverse method. See Example 23-15.

    >>> word = Text('forward')
    >>> word  
    Text('forward')
    >>> word.reverse()  
    Text('drawrof')
    >>> Text.reverse(Text('backward'))  
    Text('drawkcab')
    >>> type(Text.reverse), type(word.reverse)  
    (<class 'function'>, <class 'method'>)
    >>> list(map(Text.reverse, ['repaid', (10, 20, 30), Text('stressed')]))  
    ['diaper', (30, 20, 10), Text('desserts')]
    >>> Text.reverse.__get__(word)  
    <bound method Text.reverse of Text('forward')>
    >>> Text.reverse.__get__(None, Text)  
    <function Text.reverse at 0x101244e18>
    >>> word.reverse  
    <bound method Text.reverse of Text('forward')>
    >>> word.reverse.__self__  
    Text('forward')
    >>> word.reverse.__func__ is Text.reverse  
    True

The repr of a Text instance looks like a Text constructor call that would make an equal instance.

The reverse method returns the text spelled backward.

A method called on the class works as a function.

Note the different types: a function and a method.

Text.reverse operates as a function, even working with objects that are not instances of Text.

Any function is a nonoverriding descriptor. Calling its __get__ with an instance retrieves a method bound to that instance.

Calling the function’s __get__ with None as the instance argument retrieves the function itself.

The expression word.reverse actually invokes Text.reverse.__get__(word), returning the bound method.

The bound method object has a __self__ attribute holding a reference to the instance on which the method was called.

The __func__ attribute of the bound method is a reference to the original function attached to the managed class.

The bound method object also has a __call__ method, which handles the actual invocation. This method calls the original function referenced in __func__, passing the __self__ attribute of the method as the first argument. That’s how the implicit binding of the conventional self argument works.

The way functions are turned into bound methods is a prime example of how descriptors are used as infrastructure in the language.

After this deep dive into how descriptors and methods work, let’s go through some practical advice about their use.

The following list addresses some practical consequences of the descriptor characteristics just described:

Use property to keep it simple

The property built-in creates overriding descriptors implementing __set__ and __get__ even if you do not define a setter method.7 The default __set__ of a property raises AttributeError: can't set attribute, so a property is the easiest way to create a read-only attribute, avoiding the issue described next.

Read-only descriptors require __set__

If you use a descriptor class to implement a read-only attribute, you must remember to code both __get__ and __set__, otherwise setting a namesake attribute on an instance will shadow the descriptor. The __set__ method of a read-only attribute should just raise AttributeError with a suitable message.⁸

Validation descriptors can work with __set__ only

In a descriptor designed only for validation, the __set__ method should check the value argument it gets, and if valid, set it directly in the instance __dict__ using the descriptor instance name as key. That way, reading the attribute with the same name from the instance will be as fast as possible, because it will not require a __get__. See the code for Example 23-3.

Caching can be done efficiently with __get__ only

If you code just the __get__ method, you have a nonoverriding descriptor. These are useful to make some expensive computation and then cache the result by setting an attribute by the same name on the instance.⁹ The namesake instance attribute will shadow the descriptor, so subsequent access to that attribute will fetch it directly from the instance __dict__ and not trigger the descriptor __get__ anymore. The @functools.cached_property decorator actually produces a nonoverriding descriptor.

Nonspecial methods can be shadowed by instance attributes

Because functions and methods only implement __get__, they are nonoverriding descriptors. A simple assignment like my_obj.the_method = 7 means that further access to the_method through that instance will retrieve the number 7—without affecting the class or other instances. However, this issue does not interfere with special methods. The interpreter only looks for special methods in the class itself, in other words, repr(x) is executed as x.__class__.__repr__(x), so a __repr__ attribute defined in x has no effect on repr(x). For the same reason, the existence of an attribute named __getattr__ in an instance will not subvert the usual attribute access algorithm.

The fact that nonspecial methods can be overridden so easily in instances may sound fragile and error prone, but I personally have never been bitten by this in more than 20 years of Python coding. On the other hand, if you are doing a lot of dynamic attribute creation, where the attribute names come from data you don’t control (as we did in the earlier parts of this chapter), then you should be aware of this and perhaps implement some filtering or escaping of the dynamic attribute names to preserve your sanity.

To close this chapter, we’ll cover two features we saw with properties that we have not addressed in the context of descriptors: documentation and handling attempts to delete a managed attribute.

The docstring of a descriptor class is used to document every instance of the descriptor in the managed class. Figure 23-4 shows the help displays for the LineItem class with the Quantity and NonBlank descriptors from Examples 23-6 and 23-7.

That is somewhat unsatisfactory. In the case of LineItem, it would be good to add, for example, the information that weight must be in kilograms. That would be trivial with properties, because each property handles a specific managed attribute. But with descriptors, the same Quantity descriptor class is used for weight and price.¹⁰

The second detail we discussed with properties, but have not addressed with descriptors, is handling attempts to delete a managed attribute. That can be done by implementing a __delete__ method alongside or instead of the usual __get__ and/or __set__ in the descriptor class. I deliberately omitted coverage of __delete__ because I believe real-world usage is rare. If you need this, please see the “Implementing Descriptors” section of the Python Data Model documentation. Coding a silly descriptor class with __delete__ is left as an exercise to the leisurely reader.

Figure 23-4. Screenshots of the Python console when issuing the commands help(LineItem.weight) and help(LineItem).

Chapter Summary

The first example of this chapter was a continuation of the LineItem examples from Chapter 22. In Example 23-2, we replaced properties with descriptors. We saw that a descriptor is a class that provides instances that are deployed as attributes in the managed class. Discussing this mechanism required special terminology, introducing terms such as managed instance and storage attribute.

In “LineItem Take #4: Automatic Naming of Storage Attributes”, we removed the requirement that Quantity descriptors were declared with an explicit storage_name, which was redundant and error prone. The solution was to implement the __set_name__ special method in Quantity, to save the name of the managed property as self.storage_name.

“LineItem Take #5: A New Descriptor Type” showed how to subclass an abstract descriptor class to share code while building specialized descriptors with some common functionality.

We then looked at the different behaviors of descriptors providing or omitting the __set__ method, making the crucial distinction between overriding and nonoverriding descriptors, a.k.a. data and nondata descriptors. Through detailed testing we uncovered when descriptors are in control and when they are shadowed, bypassed, or overwritten.

Following that, we studied a particular category of nonoverriding descriptors: methods. Console experiments revealed how a function attached to a class becomes a method when accessed through an instance, by leveraging the descriptor protocol.

To conclude the chapter, “Descriptor Usage Tips” presented practical tips, and “Descriptor Docstring and Overriding Deletion” provided a brief look at how to document descriptors.

Further Reading

Besides the obligatory reference to the “Data Model” chapter, Raymond Hettinger’s “Descriptor HowTo Guide” is a valuable resource—part of the HowTo collection in the official Python documentation.

As usual with Python object model subjects, Martelli, Ravenscroft, and Holden’s Python in a Nutshell, 3rd ed. (O’Reilly) is authoritative and objective. Martelli also has a presentation titled “Python’s Object Model,” which covers properties and descriptors in depth (see the slides and video).

For more practical examples, Python Cookbook, 3rd ed., by David Beazley and Brian K. Jones (O’Reilly), has many recipes illustrating descriptors, of which I want to highlight “6.12. Reading Nested and Variable-Sized Binary Structures,” “8.10. Using Lazily Computed Properties,” “8.13. Implementing a Data Model or Type System,” and “9.9. Defining Decorators As Classes.” The last recipe of which addresses deep issues with the interaction of function decorators, descriptors, and methods, explaining how a function decorator implemented as a class with __call__ also needs to implement __get__ if it wants to work with decorating methods as well as functions.

PEP 487—Simpler customization of class creation introduced the __set_name__ special method, and includes an example of a validating descriptor.