Module Filenames

Before we go on, let’s get more formal about module filenames. You can call modules just about anything you like, but module filenames should end in a .py suffix if you plan to import them as modules. The .py is technically optional for top-level files that will be run but not imported (i.e., for scripts), but adding it in all cases makes your files’ types more obvious, may enable Python-specific features in some text editors and file explorers, and allows you to import any of your files in the future (recall that this is one way to run a file).

Because module names become variable names inside a Python program (without the .py), they should also follow the normal variable name rules outlined in Chapter 11. For instance, you can create a module file named if.py, but you cannot import it because if is a reserved word—when you try to run import if, you’ll get a syntax error. In fact, both the names of module files and the names of directories used in package imports (discussed in the next chapter) must conform to the rules for variable names presented in Chapter 11; they may, for example, contain only letters, digits, and underscores. Package directories also cannot contain platform-specific syntax such as spaces in their names.

When a module is imported, Python maps the module name to an external filename by adding a directory path from the module search path of the last chapter to the front, and a .py or other extension at the end. For instance, a module named M ultimately maps to an external file directory/M.extension that contains the module’s code.

Other Kinds of Modules

As mentioned in the preceding chapter, it is also possible to create a Python module by writing code in an external language such as C, C++, and others (e.g., Java, in the Jython implementation of the language). Such modules are called extension modules, and they are generally used to wrap up external libraries for use in Python scripts or optimize parts of a program. When imported by Python code, extension modules look and feel the same as modules coded as Python source code files—they are accessed with import statements, and they provide functions and objects as module attributes. They’re also beyond the scope of this book; see Python’s standard manuals for more details.

The import Statement

In the first example that follows, the name module1 serves two different purposes—it identifies an external file to be loaded, and it becomes a variable in the script, which references the module object after the import. In a REPL:

>>> import module1                         # Get module as a whole (one or more)
>>> module1.printer('Hello world!')        # Qualify to get names
Hello world!

The import statement simply lists one or more names of modules to load, separated by commas. Because it gives a name that refers to the whole module object, we must go through the module name to fetch its attributes (e.g., module1.printer).

The from Statement

By contrast, because from copies specific names from one file over to another scope, it allows us to use the copied names directly in the script without going through the module (e.g., printer):

>>> from module1 import printer            # Copy out a variable (one or more)
>>> printer('Hello world!')                # No need to qualify name (and can't!)
Hello world!

This form of from allows us to list one or more names to be copied out, separated by commas. Here, it has the same effect as the prior example, but because the imported name is copied into the scope where the from statement appears, using that name in the script requires less typing—we can use it directly instead of naming the enclosing module. In fact, we must; from doesn’t assign the name of the module itself, so module1 is undefined.

As you’ll see in more detail later, the from statement is really just a minor extension to the import statement—it imports the module file as usual (running the full three-step procedure of the preceding chapter), but adds an extra step that copies one or more names (not objects) out of the file. The entire file is loaded, but you’re given names for more direct access to its parts.

The from * Statement

Finally, the next example uses a special form of from: when we use a * instead of specific names, we get copies of all names assigned at the top level of the referenced module. Here again, we can then use the copied name printer in our script without going through the module name:

>>> from module1 import *                   # Copy out _all_ variables
>>> printer('Hello world!')                 # Use names unqualified
Hello world!

Technically, both import and from statements invoke the same import operation; the from * form simply adds an extra step that copies all the names in the module into the importing scope. It essentially merges one module’s namespace into another, which again means less typing for us, albeit at the expense of name segregation.

Note that only a single * works in this context; you can’t use arbitrary pattern matching to select a subset of names (you could in principle by looping through a module’s __dict__ discussed ahead, but it’s substantially more work). Also, note the from * may not really get “all” names in module if that module uses special tricks like _X naming or an __all__ list to hide some of its names from this statement, but we’ll defer to Chapter 25 for more on such tools.

And that’s it—apart from the search-path configurations of the prior chapter, modules really are simple to use. To give you a better understanding of what really happens when you define and use modules, though, let’s move on to look at some of their properties in more detail.

Imports Happen Only Once

One of the most common questions people seem to ask when they start using modules is, “Why won’t my imports keep working?” They often report that the first import works fine, but later imports during an interactive session (or program run) seem to have no effect. In fact, this is by design—modules are loaded and run on the first import or from, and only the first. Because importing is an expensive operation, by default Python does it just once per file, per process. Later import operations simply fetch the already loaded module object.

print('hello')
flag = 1                   # Initialize variable – just once!

$ python3
>>> import init            # First import: loads and runs file's code
hello
>>> init.flag              # Assignment makes an attribute
1

>>> init.flag = 2          # Change attribute in module
>>> import init            # Just fetches already loaded module
>>> init.flag              # Code wasn't rerun: attribute unchanged
2

Imports are Runtime Assignments

Just like def, import and from are executable statements, not compile-time declarations. They may be nested in if tests, to select among module options; appear in function defs, to be loaded only on calls (subject to the preceding note); be used in try statements, to provide defaults on errors; and so on. As an abstract example:

if sometest:
    from moduleA import name
else 
    from moduleB import name

Wherever they appear, imports are not resolved or run until Python reaches them while executing your program. As one upshot, imported modules and names are not available until their associated import or from statements run.

x = 1
y = [1, 2]

$ python3
>>> from share import x, y         # Copy two names out
>>> x = 'hack'                     # Changes local x only
>>> y[0] = 'hack'                  # Changes shared mutable in place
>>> x, y                           # This module's x and y
('hack', ['hack', 2])

>>> import share                   # Get module name (from doesn't)
>>> share.x                        # share's x is not my x
1
>>> share.y                        # But we share a changed mutable
['hack', 2]

$ python3
>>> from share import x, y         # Copy two names out
>>> x = 23                         # Changes my x only

>>> import share                   # Get module name
>>> share.x = 23                   # Changes x in other module

import and from Equivalence

Notice in the prior example that we have to execute an import statement after the from to access the share module name at all. from only copies names from one module to another; it does not assign the module name itself. It may help to remember that, at least conceptually, a from statement like this one:

from module import name1, name2     # Copy these two names out (only)

is equivalent to this statement sequence:

import module                       # Fetch the module object
name1 = module.name1                # Copy names out by assignment
name2 = module.name2
del module                          # Get rid of the module name – here only

Like all assignments, the from statement creates new variables in the importer, which initially refer to objects of the same names in the imported file. Only the names are copied out, though, not the objects they reference, and not the name of the module itself. When we use the from * form of this statement (from module import *), the equivalence is the same, but all the top-level names in the module are copied over to the importing scope this way.

Importantly, the first step of the from runs a normal import operation, with all the semantics outlined in the preceding chapter. As a result, the from always imports the entire module into memory if it has not yet been imported, regardless of how many names it copies out of the file. There is no way to load just part of a module file (e.g., just one function), but because modules are bytecode in standard Python instead of machine code, the performance implications are generally negligible.

Potential Pitfalls of the from Statement

One downside of the from statement is that it makes the meaning of a variable more obscure: name is less useful to the reader than module.name, and may require a search for the from that loaded it. Because of this, some Python users recommend coding import instead of from most of the time. Like most advice, though, this doesn’t always make sense. from is commonly and widely used, without dire consequences. Moreover, it’s convenient not to have to type a module’s name every time you wish to use one of its tools, especially when working in the REPL.

It is true that the from statement has the potential to corrupt namespaces, at least in principle—if you use it to import variables that happen to have the same names as existing variables in your scope, your variables will be silently overwritten. This problem doesn’t occur with the import statement because you must always go through a module’s name to get to its contents: module.attr will not clash with a variable named attr in your scope. As long as you understand that this can happen when using from, though, this isn’t a concern in practice: assigning a variable with from has the same effect as any other assignment in your code.

On the other hand, the from statement has legitimate issues when used in conjunction with the reload call, as imported names might reference prior versions of objects. Moreover, the from * form really can trash namespaces and make code difficult to understand, especially when applied to more than one file—in this case, there is no way to tell which module a name came from, short of searching external files. In effect, the from * form collapses one namespace into another, and so defeats the namespace partitioning purpose of modules. We’ll save demos of these issues for the “Module Gotchas” at the end of this part of the book, and meet tools that can minimize the from * damage with data hiding in Chapter 25.

Probably the best real-world advice here is to generally prefer import to from for simple modules; to explicitly list the variables you want in most from statements; and to limit the from * form to just one per file, or the REPL’s throw-away code. That way, any names not called out by attribute qualification or from lists can be assumed to live in the sole module imported by the from *. Some care is required when using the from statement, but armed with a little knowledge, most programmers find it to be a convenient way to access modules.

# M.py
def func():
    …do something…

# N.py
def func():
    …do something else…

# O.py
from M import func
from N import func             # This overwrites the one we fetched from M
func()                         # Calls N.func only!

# O.py
import M, N                    # Get the whole modules, not their names
M.func()                       # We can call both names now
N.func()                       # The module names make them unique

# O.py
from M import func as mfunc    # Rename uniquely with "as"
from N import func as nfunc
mfunc(); nfunc()               # Call one or the other

How Files Generate Namespaces

We’ve seen that files morph into namespaces, but how does this actually happen? The short answer is that every name that is assigned a value at the top level of a module file (i.e., not nested in a function or class body) becomes an attribute of that module.

For instance, given an assignment statement such as X = 1 at the top level of a module file M.py, the name X becomes an attribute of M, which we can refer to from outside the module as M.X. The name X also becomes a global variable to other code inside M.py, but we need to firm up the relationship of module loading and scopes to understand why:

• Module statements run on the first import. The first time a module is imported anywhere in a system, Python creates an empty module object and executes the statements in the module file one after another, from the top of the file to the bottom.

• Top-level assignments create module attributes. During an import, statements at the top level of the file not nested in a def or class that assign names (e.g., =, def) create attributes of the module object. Names assigned by these statements are stored in the module’s namespace.

• Module namespaces are dictionaries. Module namespaces created by imports may be accessed through the built-in __dict__ dictionary attribute of module objects, and may be inspected with the dir function. dir is the same as the sorted keys of __dict__ for modules, but includes inherited names for classes.

• Modules are a single scope (local is global). As we saw in Chapter 17, names at the top level of a module follow the same scope rules as names in a function, but the local and global scopes are the same—or, more formally, they follow the LEGB scope rule presented in Chapter 17, but without the L and E lookup layers.

• Module scopes outlive imports. A module’s global scope becomes an attribute dictionary of a module object after the module has been loaded. Unlike function scopes, where the local namespace exists only while a function call runs, a module file’s scope lives on after the import, providing a source of tools to importers.

Here’s a demonstration of these ideas. Suppose we create the module file in Example 23-4 with a text editor, and call it spaces.py.

print('starting to load...')
import sys
var = 23

def func(): pass

class klass: pass

print('done loading.')

The first time this module is imported (or run as a program), Python executes its statements from top to bottom. Some statements create names in the module’s namespace as a side effect, but others do actual work while the import is going on. For instance, the two print statements in this file execute at import time:

>>> import spaces
starting to load...
done loading.

Once the module is loaded, its scope becomes an attribute namespace in the module object we get back from import. We can then access attributes in this namespace by qualifying them with the name of the enclosing module:

>>> spaces.sys
<module 'sys' (built-in)>
>>> spaces.var
23
>>> spaces.func
<function func at 0x101ce7b00>
>>> spaces.klass
<class 'spaces.klass'>

Here, sys, var, func, and klass were all assigned while the module’s statements were being run, so they are attributes after the import. We’ll study classes in Part VI, but notice the sys attribute—import statements assign module objects to names, and any type of assignment to a name at the top level of a file generates a module attribute.

Namespace Dictionaries: __dict__

In fact, internally, module namespaces are stored as dictionary objects. These are just normal dictionaries with all the usual methods. When needed, we can access a module’s namespace dictionary through the module’s __dict__ attribute—for instance, to write tools that list module content generically, as we will in Chapter 25. Continuing the prior section’s example:

>>> spaces.__dict__.keys()
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', 
'__file__', '__cached__', '__builtins__', 'sys', 'var', 'func', 'klass'])

The names we assigned in the module file become dictionary keys internally, so some of the keys here reflect top-level assignments in our file. The value of var, for example, can be had two ways (though the first is normal):

>>> spaces.var, spaces.__dict__['var']
(23, 23)

Python also adds some useful names in the module’s namespace. For instance, __file__ gives the name of the file from which the module was loaded (useful to find resources bundled with code), and __name__ gives its name as known to importers (without the .py extension and directory path):

>>> spaces.__name__, spaces.__file__
('spaces', '/Users/me/…/LP6E/Chapter23/spaces.py')

To see just the names your code assigns, filter out double-underscore names as we did in Chapter 15’s dir coverage and Chapter 17’s built-in scope coverage, but this time with a generator expression. We can also omit keys because dictionaries generate keys automatically, and the dir built-in for modules is just the sorted keys list of __dict__:

>>> list(name for name in spaces.__dict__ if not name.startswith('__'))
['sys', 'var', 'func', 'klass']
>>> dir(spaces) == sorted(spaces.__dict__)
True

As another lesser-used alternative, Python’s vars built-in function simply fetches the __dict__ of a module passed to it (an option, perhaps, after you’ve written enough Python code to wear out your keyboard’s underscore key?):

>>> vars(spaces) == spaces.__dict__
True
>>> dir(spaces) == sorted(vars(spaces))
True

See Chapter 7 for other vars roles. You’ll see similar __dict__ dictionaries on class-related objects in Part VI too. In all cases, attribute fetch is similar to dictionary indexing, though only the former kicks off inheritance in classes.

Attribute Name Qualification

Speaking of attribute fetch, now that you’re becoming more familiar with modules, we should firm up the notion of name qualification more formally too. In Python, you can access the attributes of any object that has attributes using the qualification (a.k.a. attribute fetch) syntax object.attribute.

Qualification is really an expression that returns the value assigned to an attribute name associated with an object. For example, the expression spaces.sys in the previous example fetches the value assigned to sys in spaces. Similarly, if we have a built-in list object L, L.append returns the append method object associated with that list.

It’s important to keep in mind that attribute qualification has nothing to do with the scope rules we studied in Chapter 17; it’s an independent concept. When you use qualification to access names, you give Python an explicit object from which to fetch the specified names. The LEGB scope rule applies only to bare, unqualified names; it may be used for the leftmost name in a qualification path, but later names after dots search specific objects instead.

This distinction may seem blurred by the fact that module scopes morph into attributes at the end of an import, but thereafter, names are part of a module object. For reference, here are the full rules for name resolution in Python:

Simple variables

X means search for the name X in the current scopes (following the LEGB rule of Chapter 17).

Qualification

X.Y means find X in the current scopes, then search for the attribute Y in the object X (not in scopes).

Qualification paths

X.Y.Z means look up the name Y in the object X, then look up Z in the object X.Y.

Generality

Qualification works on all objects with attributes: modules, classes, C extension types, etc.

As noted earlier, in Part VI you’ll see that attribute qualification means a bit more for classes—it’s also the place where inheritance happens. In general, though, the rules outlined here apply to all names in Python.

Imports Versus Scopes

As we’ve seen, it is never possible to access names defined in another module file without first importing that file. That is, you never automatically get to see names in another file, regardless of the structure of imports or function calls in your program. A variable’s meaning is always determined by the locations of assignments in your source code, and attributes are always requested of an object explicitly.

For example, consider the following two simple modules. The first, lex1.py in Example 23-5 defines a variable X global to code in its file only, along with a function that changes the global X in this file.

X = 88                        # My X: global to this file only

def f():
    global X                  # Change this file's X
    X = 99                    # Cannot see names in other modules

The second module, lex2.py in Example 23-6, defines its own global variable X and imports and calls the function in the first module.

X = 11                        # My X: global to this file only

import lex1                   # Gain access to names in lex1
lex1.f()                      # Sets lex1.X, not this file's X
print(X, lex1.X)

When run, lex1.f changes the X in lex1, not the X in lex2. The global scope for lex1.f is always the file enclosing it, regardless of which module it is ultimately called from:

$ python3 lex2.py
11 99

In other words, import operations never give upward visibility to code in imported files—an imported file cannot see names in the importing file. More formally:

• Functions can never see names in other functions unless they are physically enclosing.

• Module code can never see names in other modules unless they are explicitly imported.

Such behavior is part of the lexical scoping notion—in Python, the scopes surrounding a piece of code are completely determined by the code’s physical position in your file. Scopes are never influenced by function calls or module imports. Some languages act differently and provide for dynamic scoping, where scopes really may depend on runtime calls. This tends to make code trickier, though, because the meaning of a variable can differ over time. In Python, scopes more simply correspond to the text of your program.

Namespace Nesting

Finally, although imports do not nest namespaces upward, they do in some sense nest downward. That is, although an imported module never has direct access to names in a file that imports it, using attribute qualification paths it is possible to descend into arbitrarily nested modules and access their attributes. For example, consider the next three files. First, nest3.py of Example 23-7, defines a single global name and attribute by assignment.

X = 3

Next, nest2.py of Example 23-8 defines its own X, then imports nest3 and uses name qualification to access the imported module’s attribute.

X = 2
import nest3

print(X, end=' ')             # My global X
print(nest3.X)                # nest3's X

And at the top, nest1.py of Example 23-9 also defines its own X, then imports nest2, and fetches attributes in both the first and second files.

X = 1
import nest2

print(X, end=' ')             # My global X
print(nest2.X, end=' ')       # nest2's X
print(nest2.nest3.X)          # Nested nest3's X

Really, when nest1 imports nest2 here, it sets up a two-level namespace nesting. By using the path of names nest2.nest3.X, it can descend into nest3, which is nested in the imported nest2. The net effect is that nest1 can see the Xs in all three files, and hence has access to all three global scopes:

$ python3 nest1.py
2 3
1 2 3

The reverse, however, is not true: nest3 cannot see names in nest2, and nest2 cannot see names in nest1. This example may be easier to grasp if you don’t think in terms of namespaces and scopes, but instead focus on the objects involved. Within nest1, nest2 is just a name that refers to an object with attributes, some of which may refer to other objects with attributes (import is an assignment). For paths like nest2.nest3.X, Python simply evaluates from left to right, fetching attributes from objects along the way.

Note that nest1 can say import nest2, and then nest2.nest3.X, but it cannot say import nest2.nest3—this syntax invokes something called package (directory) imports, the topic we’ll take up in the next chapter after we study reloads here. Package imports also create module namespace nesting, but their import statements are taken to reflect directory trees, not simple file import chains.

reload Basics

Unlike import and from:

• reload is a function in Python, not a statement.

• reload is passed an existing module object, not a string name.

• reload lives in a standard library module and must be imported itself.

Because reload expects an object, a module must have been previously imported successfully before you can reload it (and if the import was unsuccessful due to a syntax or other error, you may need to repeat it before you can reload the module). Furthermore, the syntax of import statements and reload calls differs: as a function reloads require parentheses, but import statements do not. Abstractly, reloading looks like this:

import module                     # Initial module import
…use module.attributes… 
…change the module file…

from importlib import reload      # Get reload itself
reload(module)                    # Get updated module
…use module.attributes…

The typical usage pattern is that you import a module, then change its source code in a text editor, and then reload it. This can occur when working interactively, but also in larger programs that reload periodically.

When you call reload, Python rereads the module’s code and reruns its top-level statements. Perhaps the most important thing to know about reload is that it changes a module object in place; it does not delete and re-create the module object. Because of that, every reference to an entire module object anywhere in your program is automatically affected by a reload. Here are the details:

• reload runs a module file’s new code in the module’s current namespace. Rerunning a module file’s code overwrites its existing namespace, rather than deleting and re-creating it.

• Top-level assignments in the file replace names with new values. For instance, rerunning a def statement replaces the prior version of the function in the module’s namespace by reassigning the function name.

• Reloads impact all clients that use import to fetch modules. Because clients that use import qualify to fetch attributes, they’ll find new values in the module object after a reload.

• Reloads impact future from clients only. Clients that used from to fetch attributes in the past won’t be affected by a reload; they’ll still have references to the old objects fetched before the reload.

• Reloads apply to a single module only. You must run them on each module you wish to update unless you use code or tools that apply reloads transitively.

reload Example

To demonstrate, here’s a more concrete example of reload in action. In the following, we’ll change and reload a module file without stopping the interactive Python session. Reloads are useful in other scenarios, too, but we’ll keep things simple for illustration here. First, in the text editor of your choice, write a module file named changer.py with the contents given in Example 23-10.

message = 'First version'
def printer():
    print(message)

This module creates and exports two names—one bound to a string, and another to a function. Now, start the Python interpreter (i.e., your local REPL), import the module, and call the function it exports. The function will print the value of the global message variable as you probably expect:

$ python3
>>> import changer
>>> changer.printer()
First version

Keeping the interpreter active, now edit and save the module file in another window. Change the global message variable, as well as the printer function body:

message = 'After editing'
def printer():
    print('reloaded:', message)

Then, return to the Python window and reload the module to fetch the new code. Notice in the following interaction that importing the module again has no effect; we get the original message, even though the file’s been changed. We have to call reload in order to get the new version:

>>> import changer
>>> changer.printer()                 # No effect: uses loaded module
First version

>>> from importlib import reload
>>> reload(changer)                   # Forces new code to load/run
<module 'changer' from '/…/LP6E/Chapter23/changer.py'>

>>> changer.printer()                 # Runs the new version now
reloaded: After editing

Notice that reload actually returns the module object for us—its result is usually ignored, but because expression results are printed at the interactive prompt, Python shows a default <module …> representation.

You can also reload a module by string name using the sys.modules dictionary demoed in the prior chapter if you don’t have a variable assigned to it by import:

>>> import sys
>>> reload(sys.modules['changer'])

In this case, it’s probably easier to run import changer to get a handle on the module, but the sys.modules scheme may be useful in programs that need to reload modules more generically. It’s also possible to delete modules from sys.modules to force a reload, but this is generally discouraged for reasons we’ll skip here; use reload.

reload Odds and Ends

Finally, four brief module-reload notes in closing:

• If you use reload, you’ll probably want to pair it with import instead of from, as names fetched with the latter are not updated by reload operations—leaving your names in a state that’s strange enough to warrant postponing elaboration until this part’s “gotchas” at the end of Chapter 25.

• By itself, reload updates only a single module, but it’s straightforward to code a function that applies it transitively to related modules—an extension we’ll save for a case study near the end of Chapter 25.

• Some development tools (e.g. Jupyter notebooks) have REPLs that offer an auto-reload mode that may obviate manual reload calls. This works only in specific tools, though, and doesn’t address customization roles.

• And last, reload has had a long and checkered past—morphing from built-in function, to imp module attribute in this book’s prior edition, to its current importlib host. While this may be a symptom of Python’s constant morph which is apt to relocate reload again, it must be asked: does this function have trouble working with others?!