Step 1: Data-Driven Attribute Creation

Example 22-8 shows the doctest to guide this first step.

    >>> records = load(JSON_PATH)  
    >>> speaker = records['speaker.3471']  
    >>> speaker  
    <Record serial=3471>
    >>> speaker.name, speaker.twitter  
    ('Anna Martelli Ravenscroft', 'annaraven')

load a dict with the JSON data.

The keys in records are strings built from the record type and serial number.

speaker is an instance of the Record class defined in Example 22-9.

Fields from the original JSON can be retrieved as Record instance attributes.

The code for schedule_v1.py is in Example 22-9.

import json

JSON_PATH = 'data/osconfeed.json'

class Record:
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)  

    def __repr__(self):
        return f'<{self.__class__.__name__} serial={self.serial!r}>'  

def load(path=JSON_PATH):
    records = {}  
    with open(path) as fp:
        raw_data = json.load(fp)  
    for collection, raw_records in raw_data['Schedule'].items():  
        record_type = collection[:-1]  
        for raw_record in raw_records:
            key = f'{record_type}.{raw_record["serial"]}' 
            records[key] = Record(**raw_record)  
    return records

This is a common shortcut to build an instance with attributes created from keyword arguments (detailed explanation follows).

Use the serial field to build the custom Record representation shown in Example 22-8.

load will ultimately return a dict of Record instances.

Parse the JSON, returning native Python objects: lists, dicts, strings, numbers, etc.

Iterate over the four top-level lists named 'conferences', 'events', 'speakers', and 'venues'.

record_type is the list name without the last character, so speakers becomes speaker. In Python ≥ 3.9 we can do this more explicitly with collection.removesuffix('s')—see PEP 616—String methods to remove prefixes and suffixes.

Build the key in the format 'speaker.3471'.

Create a Record instance and save it in records with the key.

The Record.__init__ method illustrates an old Python hack. Recall that the __dict__ of an object is where its attributes are kept—unless __slots__ is declared in the class, as we saw in “Saving Memory with __slots__”. So, updating an instance __dict__ with a mapping is a quick way to create a bunch of attributes in that instance.⁷

The definition of Record in Example 22-9 is so simple that you may be wondering why I did not use it before, instead of the more complicated FrozenJSON. There are two reasons. First, FrozenJSON works by recursively converting the nested mappings and lists; Record doesn’t need that because our converted dataset doesn’t have mappings nested in mappings or lists. The records contain only strings, integers, lists of strings, and lists of integers. Second reason: FrozenJSON provides access to the embedded __data dict attributes—which we used to invoke methods like .keys()—and now we don’t need that functionality either.

After reorganizing the schedule dataset, we can enhance the Record class to automatically retrieve venue and speaker records referenced in an event record. We’ll use properties to do that in the next examples.

The goal of this next version is: given an event record, reading its venue property will return a Record. This is similar to what the Django ORM does when you access a ForeignKey field: instead of the key, you get the linked model object.

We’ll start with the venue property. See the partial interaction in Example 22-10 as an example.

    >>> event = Record.fetch('event.33950')  
    >>> event  
    <Event 'There *Will* Be Bugs'>
    >>> event.venue  
    <Record serial=1449>
    >>> event.venue.name  
    'Portland 251'
    >>> event.venue_serial  
    1449

The Record.fetch static method gets a Record or an Event from the dataset.

Note that event is an instance of the Event class.

Accessing event.venue returns a Record instance.

Now it’s easy to find out the name of an event.venue.

The Event instance also has a venue_serial attribute, from the JSON data.

Event is a subclass of Record adding a venue to retrieve linked records, and a specialized __repr__ method.

The code for this section is in the schedule_v2.py module in the Fluent Python code repository. The example has nearly 60 lines, so I’ll present it in parts, starting with the enhanced Record class.

import inspect  
import json

JSON_PATH = 'data/osconfeed.json'

class Record:

    __index = None  

    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

    def __repr__(self):
        return f'<{self.__class__.__name__} serial={self.serial!r}>'

    @staticmethod  
    def fetch(key):
        if Record.__index is None:  
            Record.__index = load()
        return Record.__index[key]  

inspect will be used in load, listed in Example 22-13.

The __index private class attribute will eventually hold a reference to the dict returned by load.

fetch is a staticmethod to make it explicit that its effect is not influenced by the instance or class on which it is called.

Populate the Record.__index, if needed.

Use it to retrieve the record with the given key.

Now we get to the use of a property in the Event class, listed in Example 22-12.

class Event(Record):  

    def __repr__(self):
        try:
            return f'<{self.__class__.__name__} {self.name!r}>'  
        except AttributeError:
            return super().__repr__()

    @property
    def venue(self):
        key = f'venue.{self.venue_serial}'
        return self.__class__.fetch(key)  

Event extends Record.

If the instance has a name attribute, it is used to produce a custom representation. Otherwise, delegate to the __repr__ from Record.

The venue property builds a key from the venue_serial attribute, and passes it to the fetch class method, inherited from Record (the reason for using self.__class__ is explained shortly).

The second line of the venue method of Example 22-12 returns self​.__class__.fetch(key). Why not simply call self.fetch(key)? The simpler form works with the specific OSCON dataset because there is no event record with a 'fetch' key. But, if an event record had a key named 'fetch', then within that specific Event instance, the reference self.fetch would retrieve the value of that field, instead of the fetch class method that Event inherits from Record. This is a subtle bug, and it could easily sneak through testing because it depends on the dataset.

If the Record class behaved more like a mapping, implementing a dynamic __getitem__ instead of a dynamic __getattr__, there would be no risk of bugs from overwriting or shadowing. A custom mapping is probably the Pythonic way to implement Record. But if I took that road, we’d not be studying the tricks and traps of dynamic attribute programming.

The final piece of this example is the revised load function in Example 22-13.

def load(path=JSON_PATH):
    records = {}
    with open(path) as fp:
        raw_data = json.load(fp)
    for collection, raw_records in raw_data['Schedule'].items():
        record_type = collection[:-1]  
        cls_name = record_type.capitalize()  
        cls = globals().get(cls_name, Record)  
        if inspect.isclass(cls) and issubclass(cls, Record):  
            factory = cls  
        else:
            factory = Record  
        for raw_record in raw_records:  
            key = f'{record_type}.{raw_record["serial"]}'
            records[key] = factory(**raw_record)  
    return records

So far, no changes from the load in schedule_v1.py (Example 22-9).

Capitalize the record_type to get a possible class name; e.g., 'event' becomes 'Event'.

Get an object by that name from the module global scope; get the Record class if there’s no such object.

If the object just retrieved is a class, and is a subclass of Record…

…bind the factory name to it. This means factory may be any subclass of Record, depending on the record_type.

Otherwise, bind the factory name to Record.

The for loop that creates the key and saves the records is the same as before, except that…

…the object stored in records is constructed by factory, which may be Record or a subclass like Event, selected according to the record_type.

Note that the only record_type that has a custom class is Event, but if classes named Speaker or Venue are coded, load will automatically use those classes when building and saving records, instead of the default Record class.

We’ll now apply the same idea to a new speakers property in the Events class.

The name of the venue property in Example 22-12 does not match a field name in records of the "events" collection. Its data comes from a venue_serial field name. In contrast, each record in the events collection has a speakers field with a list of serial numbers. We want to expose that information as a speakers property in Event instances, which returns a list of Record instances. This name clash requires some special attention, as Example 22-14 reveals.

    @property
    def speakers(self):
        spkr_serials = self.__dict__['speakers']  
        fetch = self.__class__.fetch
        return [fetch(f'speaker.{key}')
                for key in spkr_serials]  

The data we want is in a speakers attribute, but we must retrieve it directly from the instance __dict__ to avoid a recursive call to the speakers property.

Return a list of all records with keys corresponding to the numbers in spkr_serials.

Inside the speakers method, trying to read self.speakers will invoke the property itself, quickly raising a RecursionError. However, if we read the same data via self.__dict__['speakers'], Python’s usual algorithm for retrieving attributes is bypassed, the property is not called, and the recursion is avoided. For this reason, reading or writing data directly to an object’s __dict__ is a common Python metaprogramming trick.

As I coded the list comprehension in Example 22-14, my programmer’s lizard brain thought: “This may be expensive.” Not really, because events in the OSCON dataset have few speakers, so coding anything more complicated would be premature optimization. However, caching a property is a common need—and there are caveats. So let’s see how to do that in the next examples.

Step 4: Bespoke Property Cache

Caching properties is a common need because there is an expectation that an expression like event.venue should be inexpensive.8 Some form of caching could become necessary if the Record.fetch method behind the Event properties needed to query a database or a web API.

In the first edition Fluent Python, I coded the custom caching logic for the speakers method, as shown in Example 22-15.

    @property
    def speakers(self):
        if not hasattr(self, '__speaker_objs'):  
            spkr_serials = self.__dict__['speakers']
            fetch = self.__class__.fetch
            self.__speaker_objs = [fetch(f'speaker.{key}')
                    for key in spkr_serials]
        return self.__speaker_objs  

If the instance doesn’t have an attribute named __speaker_objs, fetch the speaker objects and store them there.

Return self.__speaker_objs.

The handmade caching in Example 22-15 is straightforward, but creating an attribute after the instance is initialized defeats the PEP 412—Key-Sharing Dictionary optimization, as explained in “Practical Consequences of How dict Works”. Depending on the size of the dataset, the difference in memory usage may be important.

A similar hand-rolled solution that works well with the key-sharing optimization requires coding an __init__ for the Event class, to create the necessary __speaker_objs initialized to None, and then checking for that in the speakers method. See Example 22-16.

class Event(Record):

    def __init__(self, **kwargs):
        self.__speaker_objs = None
        super().__init__(**kwargs)

# 15 lines omitted...
    @property
    def speakers(self):
        if self.__speaker_objs is None:
            spkr_serials = self.__dict__['speakers']
            fetch = self.__class__.fetch
            self.__speaker_objs = [fetch(f'speaker.{key}')
                    for key in spkr_serials]
        return self.__speaker_objs

Examples 22-15 and 22-16 illustrate simple caching techniques that are fairly common in legacy Python codebases. However, in multithreaded programs, handmade caches like those introduce race conditions that may lead to corrupted data. If two threads are reading a property that was not previously cached, the first thread will need to compute the data for the cache attribute (__speaker_objs in the examples) and the second thread may read a cached value that is not yet complete.

Fortunately, Python 3.8 introduced the @functools.cached_property decorator, which is thread safe. Unfortunately, it comes with a couple of caveats, explained next.

The functools module provides three decorators for caching. We saw @cache and @lru_cache in “Memoization with functools.cache” (Chapter 9). Python 3.8 introduced @cached_property.

The functools.cached_property decorator caches the result of the method in an instance attribute with the same name. For example, in Example 22-17, the value computed by the venue method is stored in a venue attribute in self. After that, when client code tries to read venue, the newly created venue instance attribute is used instead of the method.

    @cached_property
    def venue(self):
        key = f'venue.{self.venue_serial}'
        return self.__class__.fetch(key)

In “Step 3: Property Overriding an Existing Attribute”, we saw that a property shadows an instance attribute by the same name. If that is true, how can @cached_property work? If the property overrides the instance attribute, the venue attribute will be ignored and the venue method will always be called, computing the key and running fetch every time!

The answer is a bit sad: cached_property is a misnomer. The @cached_property decorator does not create a full-fledged property, it creates a nonoverriding descriptor. A descriptor is an object that manages the access to an attribute in another class. We will dive into descriptors in Chapter 23. The property decorator is a high-level API to create an overriding descriptor. Chapter 23 will include a through explanation about overriding versus nonoverriding descriptors.

For now, let us set aside the underlying implementation and focus on the differences between cached_property and property from a user’s point of view. Raymond Hettinger explains them very well in the Python docs:

The mechanics of cached_property() are somewhat different from property(). A regular property blocks attribute writes unless a setter is defined. In contrast, a cached_property allows writes.

The cached_property decorator only runs on lookups and only when an attribute of the same name doesn’t exist. When it does run, the cached_property writes to the attribute with the same name. Subsequent attribute reads and writes take precedence over the cached_property method and it works like a normal attribute.

The cached value can be cleared by deleting the attribute. This allows the cached_property method to run again.⁹

Back to our Event class: the specific behavior of @cached_property makes it unsuitable to decorate speakers, because that method relies on an existing attribute also named speakers, containing the serial numbers of the event speakers.

Despite these limitations, @cached_property addresses a common need in a simple way, and it is thread safe. Its Python code is an example of using a reentrant lock.

The @cached_property documentation recommends an alternative solution that we can use with speakers: stacking @property and @cache decorators, as shown in Example 22-18.

    @property  
    @cache  
    def speakers(self):
        spkr_serials = self.__dict__['speakers']
        fetch = self.__class__.fetch
        return [fetch(f'speaker.{key}')
                for key in spkr_serials]

The order is important: @property goes on top…

…of @cache.

Recall from “Stacked Decorators” the meaning of that syntax. The top three lines of Example 22-18 are similar to:

speakers = property(cache(speakers))

The @cache is applied to speakers, returning a new function. That function then is decorated by @property, which replaces it with a newly constructed property.

This wraps up our discussion of read-only properties and caching decorators, exploring the OSCON dataset. In the next section, we start a new series of examples creating read/write properties.