Arguments and Shared References

To demo argument-passing properties at work, consider the following code:

>>> def f(a):                 # a is assigned a reference to the passed object
        a = 99                # Changes local variable a only

>>> b = 88
>>> f(b)                      # a and b both reference same 88 initially
>>> b                         # But b is not changed by assignment to a in f
88

In this example, the variable a is assigned the object 88 at the moment the function is called with f(b), but a lives only within the called function’s local scope. Changing a inside the function has no effect on the place where the function is called; it simply resets the local variable a to a completely different object, 99.

That’s what is meant by a lack of name aliasing—assignment to an argument name inside a function (e.g., a=99) does not magically change a variable like b in the scope of the function call. Argument names may share passed objects initially (they are essentially pointers to those objects), but only temporarily, when the function is first called. As soon as an argument name is reassigned, this relationship ends.

At least, that’s the case for assignment to argument names themselves. When arguments are passed mutable objects like lists and dictionaries, we also need to be aware that in-place changes to such objects may live on after a function exits, and hence impact callers. Here’s an example that demonstrates this behavior:

>>> def changer(a, b):        # Arguments assigned references to objects
        a = 2                 # Changes local name's value only
        b[0] = 'mod'          # Changes shared object in place: outlives call

>>> X = 1
>>> L = [1, 2]                # Caller:
>>> changer(X, L)             # Pass immutable and mutable objects
>>> X, L                      # X is unchanged, but L is different!
(1, ['mod', 2])

In this code, the changer function assigns values to argument a itself, and to a component of the object referenced by argument b. These two assignments within the function are only slightly different in syntax but have radically different results:

• Because a is a local variable name in the function’s scope, the first assignment has no effect on the caller—it simply changes the local variable a to reference a completely different object, and does not change the value of the name X in the caller’s scope. This is the same as in the prior example.

• Argument b is a local variable name, too, but it is passed a mutable object—the list that L also references in the caller’s scope. Because the assignment to b[0] in the function is an in-place change to a shared object, its result impacts the value of L after the function returns.

Really, the second assignment statement in changer doesn’t change b—it changes part of the object that b currently references. This in-place change impacts the caller only because the changed object outlives the function call. The name L hasn’t changed either—it still references the same, changed object—but it seems as though L differs after the call because the value it references has been modified within the function. In effect, the list name L serves as both input to and output from the function.

Figure 18-1 illustrates the name/object bindings that exist immediately after the function has been called, and before its code has run. When the call begins, two objects are shared among four names.

If this example is still confusing, it may help to notice that the effect of the automatic assignments of the passed-in arguments is the same as running a series of simple assignment statements. In terms of the first argument, the assignment has no effect on the caller:

>>> X = 1
>>> a = X               # They share the same object
>>> a = 2               # Name change resets 'a' only, 'X' is still 1
>>> X
1

Figure 18-1. Function arguments and shared object references

The assignment through the second argument does affect a variable at the call, though, because it is an in-place object change:

>>> L = [1, 2]
>>> b = L               # They share the same object
>>> b[0] = 'mod'        # In-place change from 'b': 'L' sees the change too
>>> L
['mod', 2]

If you recall our discussions about shared mutable objects in Chapters 6 and 9, you’ll recognize the phenomenon at work: changing a mutable object in place can impact other references to that object. Here, the effect is to make one of the arguments work like both an input and an output of the function.

Avoiding Mutable Argument Changes

This behavior of in-place changes to mutable arguments isn’t a bug—it’s simply the way argument passing works in Python, and turns out to be widely useful in practice. Arguments are normally passed to functions by reference because that is what we normally want. It means we can pass large objects around our programs without making multiple copies along the way, and we can easily update these objects as we go. In fact, as you’ll see in Part VI, Python’s class and OOP model depends upon changing a passed-in “self” argument in place, to update mutable object state.

If we don’t want in-place changes within functions to impact objects we pass to them, though, we can simply make explicit copies of mutable objects, as we saw in Chapter 6. For function arguments, we can always copy the list at the point of call with tools like list, list.copy, or an empty slice (dictionaries have similar copy tools):

L = [1, 2]
changer(X, L[:])        # Pass a copy, so our 'L' does not change

We can also copy within the function itself, if we never want to change passed-in objects, regardless of how the function is called:

def changer(a, b):
    b = b.copy()        # Copy input list so we don't impact caller
    a = 2
    b[0] = 'mod'        # Changes our list copy only

Both of these copying schemes don’t stop the function from changing the object—they just prevent those changes from impacting the caller. To really prevent changes, we can always convert to immutable objects to force the issue. Tuples, for example, raise an exception when changes are attempted:

L = [1, 2]
changer(X, tuple(L))    # Pass a tuple, so changes are errors
TypeError: 'tuple' object does not support item assignment

This scheme uses the built-in tuple function, which builds a new tuple out of all the items in a sequence (really, any iterable). It’s also something of an extreme—because it forces the function to be written to never change passed-in arguments, this solution might impose more limitations on the function than it should, and so should generally be avoided (you never know when changing arguments might come in handy for other calls in the future). Using this technique will also make the function lose the ability to call any list-specific methods on the argument, including methods that do not change the object in place (e.g. copy, though tuples have adopted list count and index).

The main point to remember here is that functions might update mutable objects like lists and dictionaries passed into them. This isn’t necessarily a problem if it’s expected, and often serves useful purposes. Moreover, functions that change passed-in mutable objects in place are probably designed and intended to do so—the change is likely part of a well-defined API that you shouldn’t violate by making copies.

However, you do have to be aware of this property—if objects change out from under you unexpectedly, check whether a called function might be responsible, and make copies when objects are passed if needed.

Simulating Output Parameters and Multiple Results

Here’s another function topic from the assignments department. We’ve already discussed the return statement and used it in examples. What we haven’t yet seen is a common though unusual coding technique it enables: because return can send back any sort of object, it can return multiple values by packaging them in a tuple or other collection type. In fact, although Python doesn’t support what some languages label “call by reference” argument passing, we can simulate it by returning tuples and assigning the results back to the original argument names in the caller:

>>> def multiple(x, y):
        x = 2                # Changes local names only
        y = [3, 4]
        return x, y          # Return multiple new values in a tuple

>>> X = 1
>>> L = [1, 2]
>>> X, L = multiple(X, L)    # Assign results to caller's names
>>> X, L
(2, [3, 4])

It looks like the code is returning two values here, but it’s really just one—a two-item tuple with the optional surrounding parentheses omitted. After the call returns, we can use tuple (a.k.a. sequence) assignment to unpack the parts of the returned tuple. (If you’ve forgotten why this works, flip back to “Tuples” in Chapter 4 and Chapter 9, and “Assignments” in Chapter 11.) The net effect of this coding pattern is to both send back multiple results and simulate the output parameters of other languages by explicit assignments. Here, X and L change after the call—but only because the code said so. Lists would work, too, but tuples sans () are less to type, and hence common.

Argument Matching Overview

Before we get into syntax details, it’s important to stress that these special modes are optional and deal only with matching objects to names; the underlying passing mechanism after the matching takes place is still assignment. In fact, some of these tools are intended more for people writing libraries than for application developers. That said, you may stumble across these modes even if you don’t code them yourself, so the following summarizes all the options:

Positionals: matched from left to right

The normal case, which we’ve mostly been using so far, is to match argument values (passed in a call) to argument names (listed in a function definition) by position, from left to right.

Keywords: matched by argument name

Alternatively, callers can explicitly specify which argument in the function is to receive a value, by giving the definition’s name for the argument with name=value syntax.

Defaults: specify values for optional arguments that aren’t passed

Functions themselves can specify default values for arguments to receive if the call passes too few values, again using the name=value syntax.

Starred collectors: collect arbitrarily many positional or keyword arguments

Function definitions can use special arguments preceded with one or two * characters to collect an arbitrary number of arguments after other matching. This feature is sometimes referred to as varargs, after a variable-length argument list tool in the C language; in Python, the arguments are collected in a normal object.

Starred unpackers: pass arbitrarily many positional or keyword arguments

Function calls can also use the one or two * syntax to unpack argument collections into separate arguments. This is the inverse of a * in a function definition—in the definition it means collect arbitrarily many arguments, while in the call it means unpack arbitrarily many arguments, and pass them individually as discrete values.

Keyword-only arguments: arguments that must be passed by name

Function definitions can also use a “*” in their headers to specify arguments that must be passed by name as keyword arguments, not position. This is often used for configuration options that augment primary arguments.

Positional-only arguments: arguments that must be passed by position

As of Python 3.8, function definitions may additionally use a “/” in their header’s arguments list to specify that all arguments preceding it must be passed by position, not keyword-argument name.

Argument Matching Syntax

As a reference and preview, Table 18-1 summarizes the syntax that invokes the argument-matching modes.

The argument-matching modes listed in this table break down between function calls and definitions as follows:

In a function call (the first four rows of the table)

Simple values are matched to definition arguments by position, but using the name=value form tells Python to match arguments by name instead; these are called keyword arguments. Any number of *iterable or **dict forms can be used in a call, allowing us to bundle many positional or keyword objects in iterables and mappings, respectively, and unpack them as separate, individual arguments when they are passed to the function.

In a function definition (the rest of the table)

A simple name is matched by position or name depending on how the caller passes it, but the name=value form specifies a default value. The *name form collects any extra unmatched positional arguments in a tuple, and **name collects unmatched keyword arguments in a dictionary. In addition, any normal or defaulted argument names following a *name or a bare * are keyword-only arguments and must be passed by keyword in calls, and arguments preceding a / are positional-only arguments that must not be passed by keyword name.

Of these, keyword arguments and defaults are probably the most commonly used in Python code. We’ve informally used both of these earlier in this book:

• We’ve already used keywords to specify options to the print function, but they are more general—keywords allow us to label any argument with its name, to make calls more explicit and informational.

• We met defaults earlier, too, as a way to pass in values from the enclosing function’s scope, but they are also more general—they allow us to make any argument optional, providing its default value in a function definition.

As you’ll see ahead, the combination of defaults in a function definition and keywords in a call further allows us to pick and choose which defaults to override per call.

In short, special argument-matching modes let you be fairly liberal about how many arguments must be passed to a function. If a function specifies defaults, they are used if you pass too few arguments. If a function uses the * argument-collector forms, you can seemingly pass too many arguments; the * names collect the extra arguments in data structures for processing within the function.

Argument Passing Details

If you choose to use and combine the special argument-matching modes, Python will ask you to follow some ordering rules among the modes’ optional components. We’re going to defer full and formal coverage of these until we’ve had a chance to observe these modes in action, but as some initial tips:

• In a function call, any number of positionals, keywords, and starred unpackings can be used, but positional arguments must precede keyword arguments and **dict unpackings, and *iterable unpackings must precede **dict unpackings.

• In a function definition, arguments must appear in this order: any positional-only arguments; followed by any positional-or-keyword arguments; followed by the optional *name positional collector or * and any keyword-only arguments; followed by the optional **name keyword collector. Arguments can have optional defaults (name=value), but once a default is used, all arguments must use defaults up to a *, after which keyword-only arguments allow defaults and nondefaults to be freely mixed.

If you mix arguments in any other order, you will get a syntax error because the combinations can be ambiguous. The steps that Python internally carries out to match arguments before assignment can roughly be described as follows:

1. Unpack all *args at the call into nonkeyword arguments.

2. Unpack all **args at the call into keyword arguments.

3. Assign nonkeyword arguments by position.

4. Assign keyword arguments by matching names.

5. Collect extra nonkeyword arguments in the *name tuple.

6. Collect extra keyword arguments in the **name dictionary.

7. Assign default values to unassigned arguments.

After arguments in call and definition are matched, Python checks to make sure each argument is passed just one value (or else an error is raised) and then assigns argument names to the objects passed to them and runs the function body.

The actual matching algorithm Python uses is a bit more complex, so we’ll defer to Python’s standard language manual for a more exact description. It’s not required reading, but tracing Python’s matching algorithm may help you to understand some convoluted cases, especially when modes are mixed.

We’ll return to the ordering rules in function calls in definitions with higher fidelity after we’ve had a chance to meet all the players. Let’s get started in the next section with the most common of the bunch.

Keyword and Default Examples

Argument passing is simpler in code than the preceding descriptions may imply. First off, if you don’t use any special matching syntax, Python matches names by position from left to right. For instance, if you define a function that requires three arguments, you must call it with three arguments:

>>> def f(a, b, c): print(a, b, c)

>>> f(1, 2, 3)
1 2 3

Here, we pass by position—a is matched to 1, b is matched to 2, and so on. This works like it does in most other programming languages.

>>> f(c=3, b=2, a=1)
1 2 3

>>> f(1, c=3, b=2)            # a gets 1 by position, b and c passed by name
1 2 3

func(title='Learning Python', edition=6, year=2024, python=3.12)

func(title='Learning Python', 6, 2024, 3.12)

>>> def f(a, b=2, c=3): print(a, b, c)          # a required, b and c optional

>>> f(1)                   # Use defaults
1 2 3
>>> f(a=1)
1 2 3

>>> f(1, 4)                # Override defaults
1 4 3
>>> f(1, 4, 5)
1 4 5

>>> f(1, c=6)              # Choose defaults: b in the middle
1 2 6

def func(code, hack, script=0, app=0):   # First 2 required
    print((code, hack, script, app))

func(1, 2)                               # Output: (1, 2, 0, 0)
func(1, app=1, hack=0)                   # Output: (1, 0, 0, 1)
func(code=1, hack=0)                     # Output: (1, 0, 0, 0)
func(script=1, hack=2, code=3)           # Output: (3, 2, 1, 0)
func(1, 2, 3, 4)                         # Output: (1, 2, 3, 4)

>>> func = lambda code, hack, script=0, app=0: (code, hack, script, app)
>>> func(1, 2)
(1, 2, 0, 0)
>>> func(script=1, hack=2, code=3)
(3, 2, 1, 0)

Arbitrary Arguments Examples

The last two argument-matching extensions, * and **, are designed to support any number of arguments. Both can appear in either the function definition or a function call, and they have related purposes in the two locations.

>>> def f(*args): print(args)

>>> f()
()
>>> f(1)
(1,)
>>> f(1, 2, 3, 4)
(1, 2, 3, 4)

>>> def f(**args): print(args)

>>> f()
{}
>>> f(a=1, b=2)
{'a': 1, 'b': 2}

>>> def f(a, *pargs, **kargs): print(a, pargs, kargs)

>>> f(1, 2, 3, x=1, y=2)
1 (2, 3) {'x': 1, 'y': 2}

>>> def func(a, b, c, d): print(a, b, c, d)

>>> args = (1, 2)
>>> args += (3, 4)
>>> func(*args)                            # Same as func(1, 2, 3, 4)
1 2 3 4

>>> args = {'a': 1, 'b': 2, 'c': 3}
>>> args['d'] = 4
>>> func(**args)                           # Same as func(a=1, b=2, c=3, d=4)
1 2 3 4

>>> func(*(1, 2), **{'d': 4, 'c': 3})      # Same as func(1, 2, d=4, c=3)
1 2 3 4
>>> func(1, *[2, 3], **dict(d=4))          # Same as func(1, 2, 3, d=4)
1 2 3 4
>>> func(1, c=3, *[2], **{'d': 4})         # Same as func(1, 2, c=3, d=4)
1 2 3 4
>>> func(1, *(2, 3), d=4)                  # Same as func(1, 2, 3, d=4)
1 2 3 4
>>> func(1, *[2], c=3, **dict(d=4))        # Same as func(1, 2, c=3, d=4)
1 2 3 4

>>> pargs, kargs = (1, 2), dict(d=4, c=3)
>>> func(*pargs, **kargs)
1 2 3 4

>>> def func(a, b, c, d): print(a, b, c, d)

>>> func(*[1], *(2,), **dict(c=3), **{'d': 4})         # *Positionals + **keywords
1 2 3 4
>>> func(*[1], *(2,), *[3, 4])                         # All *positionals
1 2 3 4
>>> func(**dict(a=1, b=2), **dict(c=3), **{'d': 4})    # All **keywords
1 2 3 4
>>> func(*[1], 2, **dict(c=3), d=4)                    # All call modes at once!
1 2 3 4

>>> func(*[1], **dict(b=2, c=3), *[4])
SyntaxError: iterable argument unpacking follows keyword argument unpacking

func(*range(4))            # Same as func(0, 1, 2, 3)
func(*open('filename'))    # Read+pass lines as arguments

if sometest:
    action, args = func1, (1,)             # Call func1 with one arg in this case
else:
    action, args = func2, (1, 2, 3)        # Call func2 with three args instead
…
action(*args)                              # Dispatch generically

args = (2,3)
args += (4,)
…
func3(*args)

def tracer(func, *pargs, **kargs):         # Accept arbitrary arguments
    print('calling:', func.__name__)
    return func(*pargs, **kargs)           # Pass along arbitrary arguments

def func(a, b, c, d):
    return a + b + c + d

print(tracer(func, 1, 2, c=3, d=4))

$ python3 tracer0.py
calling: func
10

Keyword-Only Arguments

Besides accepting keyword (i.e., pass-by-name) arguments in general, function definitions can also specify that some arguments must always be passed by keyword only and will never be filled in by a positional argument. This extension, known as keyword-only arguments, can be useful if we want a function to both process any number of arguments and accept possibly optional configuration options.

Syntactically, keyword-only arguments are coded as named arguments that may appear after *name in the arguments list. All such arguments must be passed using keyword syntax in the call. For example, in the following, a may be passed by name or position, b collects any extra positional arguments, and c must be passed by keyword only:

>>> def kwonly(a, *b, c): print(a, b, c)

>>> kwonly(1, 2, c=3)
1 (2,) 3
>>> kwonly(a=1, c=3)
1 () 3
>>> kwonly(1, 2, 3)
TypeError: kwonly() missing 1 required keyword-only argument: 'c'

If we don’t need to collect arbitrary positionals, we can also use a bare * character by itself in the arguments list to introduce keyword-only arguments. This indicates that a function does not accept a variable-length argument list but still expects all arguments following the * to be passed as keywords. In the next function, a may be passed by position or name again, but b and c must be keywords, and no extra positionals are allowed:

>>> def kwonly(a, *, b, c): print(a, b, c)

>>> kwonly(1, c=3, b=2)
1 2 3
>>> kwonly(c=3, b=2, a=1)
1 2 3
>>> kwonly(1, 2, 3)
TypeError: kwonly() takes 1 positional argument but 3 were given
>>> kwonly(1)
TypeError: kwonly() missing 2 required keyword-only arguments: 'b' and 'c'

You can still use defaults for keyword-only arguments, even though they appear after the * in the function. In the following, a may be passed by name or position, and b and c are optional but must be passed by keyword if used:

>>> def kwonly(a, *, b='code', c='app'): print(a, b, c)

>>> kwonly(1)
1 code app
>>> kwonly(1, c='hack')
1 code hack
>>> kwonly(a='py')
py code app
>>> kwonly(c=3, b=2, a=1)
1 2 3
>>> kwonly(1, 2)
TypeError: kwonly() takes 1 positional argument but 2 were given

In fact, keyword-only arguments with defaults are optional, but those without defaults effectively become required keywords for the function—like b in the following:

>>> def kwonly(a, *, b, c='hack'): print(a, b, c)

>>> kwonly(1, b='code')
1 code hack
>>> kwonly(1, c='code')
TypeError: kwonly() missing 1 required keyword-only argument: 'b'
>>> kwonly(1, 2)
TypeError: kwonly() takes 1 positional argument but 2 were given

As noted earlier, keyword-only arguments also allow defaults and nondefaults to be mixed, unlike their otherwise more flexible cohorts coded before the optional *—an ostensible inconsistency we’ll return to later:

>>> def kwonly(a, *, b=2, c, d=4): print(a, b, c, d)

>>> kwonly(1, c=3)
1 2 3 4
>>> kwonly(1, c=5, b=6)
1 6 5 4
>>> kwonly(1)
TypeError: kwonly() missing 1 required keyword-only argument: 'c'
>>> kwonly(1, 2, 3)
TypeError: kwonly() takes 1 positional argument but 3 were given

>>> def badmix(b=2, c, d=5): …
SyntaxError: parameter without a default follows parameter with a default

Finally, note that keyword-only arguments must be specified after a single star, not two—nothing can appear after the **args arbitrary-keywords form, and, unlike *, a ** can’t appear by itself in the arguments list. More generally, keyword-only arguments must be coded between the * and the optional **, and an argument that appears before * is a possibly default argument that can be passed by position or keyword, not keyword-only:

>>> def kwonly(a, **kargs, b, c):
SyntaxError: arguments cannot follow var-keyword argument
>>> def kwonly(a, **, b, c):
SyntaxError: invalid syntax
>>> def mixed(a, *b, **d, c=6):
SyntaxError: arguments cannot follow var-keyword argument

These failures will make more sense after we get formal about argument ordering rules later in this chapter. They may also appear to be worst cases in the artificial examples here, but they can come up in real practice, especially for people who write libraries for other Python programmers to use—which leads to the next point.

process(X, Y, Z)                    # Use flag's default
process(X, Y, notify=True)          # Override flag default

def process(*args, notify=False): …

Positional-Only Arguments

Beginning with Python 3.8, function definitions may also include a / in the arguments list to designate that all arguments preceding it (i.e., to its left) must be passed by position, not by keyword-argument name. Though arguably ad hoc on first sighting in an arguments list, this notation was being used in documentation for built-in functions that did not accept keywords; making it available to function coders as part of Python’s syntax was deemed a logical extension for library developers who may not want to expose argument names for use by clients as keywords.

To demo, the following function specifies that a and b must be passed by position, though c is more flexible:

>>> def mostlypos(a, b, /, c): print(a, b, c)
 
>>> mostlypos(1, 2, 3)
1 2 3
>>> mostlypos(1, 2, c=3)
1 2 3

Passing either of the first two arguments by keyword, however, fails:

>>> mostlypos(1, b=2, c=3)
TypeError: mostlypos() got some positional-only arguments passed as 
keyword arguments: 'b'

>>> mostlypos(c=3, b=2, a=1)
TypeError: mostlypos() got some positional-only arguments passed as 
keyword arguments: 'a, b'

To define a function that allows only positional arguments, simply code the slash at the end:

>>> def allpos(a, b, c, /): print(a, b, c)
... 
>>> allpos(1, 2, 3)
1 2 3
>>> allpos(1, 2, c=3)
TypeError: mostlypos() got some positional-only arguments passed as keyword …

As you should expect, the slash works the same in a lambda argument list:

>>> f = lambda a, b, /, c: print(a, b, c)

>>> f(1, 2, c=3)
1 2 3
>>> f(1, b=2, c=3)
TypeError: <lambda>() got some positional-only arguments passed as keyword …

And functions can combine positional- and keyword-only arguments to be as rigid as they wish:

>>> def combo(a, b, /, *, c, d): print(a, b, c, d)
... 
>>> combo(1, 2, c=3, d=4)
1 2 3 4
>>> combo(1, 2, 3, 4)
TypeError: combo() takes 2 positional arguments but 4 were given
>>> combo(a=1, b=2, c=3, d=4)
TypeError: combo() got some positional-only arguments passed as keyword …

It’s up to you to ponder whether or not use cases for this syntax justify its convolution of function definitions that follows in the next section. Given that Python users somehow got by without it for over three decades, though, this seems a tough sell. For more on the rationale and usage of the positional-only slash, see Python’s standard manuals. Also watch for an example in Chapter 21 that uses it to avoid name clashes—and may or may not be compelling.

Definition Ordering

In function definitions, argument lists are enclosed in parenthesis in a def statement and coded before a colon in a lambda expression, but follow the same format in both. In short, they consist of four optional parts that must appear in the following order, where position means a simple value in calls, and keyword means a name=value pair:

1. One or more arguments that must be passed by position only, followed by a single /

2. Any number of arguments that can be passed by either position or keyword

3. A single * by itself or a single *name positional-argument collector, optionally followed by any number of arguments that must be passed by keyword only

4. A single **name keyword-argument collector

In all cases, individual arguments, including a bare / or *, are separated by commas. In addition, any nonstarred argument name can have a name=expression default, but all names must have defaults after the first that does, up to the * (keyword-only argument runs following a star may freely mix default and nondefault names).

Inherent in this ordering, positional-only arguments must appear first, *name ends positional-argument runs and collects unmatched positionals, and **name ends the entire arguments list and collects unmatched keywords. As noted earlier, the **name collector’s keys retain the order in which keyword arguments were passed to the function.

def name(arguments-list): statements
lambda arguments-list: expression

arguments-list = 
         [positional-only-arguments, /]
         [positional-or-keyword-arguments]
         [* -or- *positional-collector, [keyword-only-arguments]] 
         [**keyword-collector]

>>> def f(a, /, b, c=3, *ps, e=4, **ks): 
        print(f'{a=}, {b=}, {c=}, {ps=}, {e=}, {ks=}')

>>> f(0, 1, 2, 3, 4, e=5, f=6, g=7)
a=0, b=1, c=2, ps=(3, 4), e=5, ks={'f': 6, 'g': 7}

>>> def f(a, *ps, /, b): pass
SyntaxError: / must be ahead of *

>>> def f(a, **ks, b): pass
SyntaxError: arguments cannot follow var-keyword argument

>>> def f(a, b, c=3, d, e): pass
SyntaxError: parameter without a default follows parameter with a default

>>> def f(a, b=2, /, c): pass
SyntaxError: parameter without a default follows parameter with a default

>>> def f(a, *, x=1, y): ...      # OK: x and y must be keywords

>>> def f(a=1, b, *, x=1): ...    # Not OK: a and b may be positional
SyntaxError: parameter without a default follows parameter with a default

Calls Ordering

On the other side of the fence, the syntax is similar but the rules differ. In function calls, all positional arguments must precede all keyword arguments, and any number of starred unpackings can be mixed in with individual values: one star unpacks into multiple positional arguments, two unpacks into keywords, and all positional arguments and one-star unpackings must precede all two-star unpackings.

In more detail, function (and by extension, method) calls consist of three optional parts in the following order:

1. Any combination of one or more expression positional arguments, and *expression iterable unpackings transformed into positional arguments

2. Any combination of one or more name=expression keyword arguments, and *expression iterable unpackings transformed into positional arguments

3. Any combination of one or more name=expression keyword arguments, and **expression mapping unpackings transformed into keyword arguments

In all cases, all arguments, starred or otherwise, are separated by commas.

function([positional-values -or- *iterable-positional-unpackings]
         [keyword-arguments -or- *iterable-positional-unpackings]
         [keyword-arguments -or- **mapping-keyword-unpackings])

>>> def f(a, b, c, d, e, f, g, h, i): 
        print(a, b, c, d, e, f, g, h, i)

>>> f(*[1], 2, *[3], 4, f=6, *[5], **dict(g=7), h=8, **{'i': 9})
1 2 3 4 5 6 7 8 9

>>> f(1, 2, c=3, 4)
SyntaxError: positional argument follows keyword argument unpacking

>>> f(1, 2, **{'d': 3}, 4)
SyntaxError: positional argument follows keyword argument unpacking
>>> f(1, 2, **{'d': 3}, *[4])
SyntaxError: iterable argument unpacking follows keyword argument unpacking

>>> f(1, 2, d=3, *[4])    # OK, but why? – like first error above
>>> f(1, 2, d=3, 4)       # Not OK, but why? – like preceding line
SyntaxError: positional argument follows keyword argument

Full Credit

The following script file, mins.py in Example 18-2, shows four ways to code this operation (some of which were suggested by students in a group exercise designed to prevent post-lunch napping):

• The first function fetches the first argument from its args tuple, and traverses the rest by slicing off the first (there’s no point in comparing the first object to itself, especially if it might be a large structure).

• The second version lets Python pick off the first and rest of the arguments automatically, and so avoids an index and slice; the code is simpler, and may be faster (though it would take many calls to matter).

• The third converts from a tuple to a list with the built-in list call and employs the list sort method: the first item has lowest value after an ascending-value sort.

• The fourth sorts too, but skips the list conversion (and two lines) by using the sorted built-in function.

Python sorting tools are coded in C, so they can be quicker than the other approaches at times, but the linear scans of the first two techniques may often make them faster.¹

"Find minimum value among all passed arguments of comparable types"

def min1(*args):
    res = args[0]
    for arg in args[1:]:
        if arg < res:
            res = arg
    return res

def min2(first, *rest):
    for arg in rest:
        if arg < first:
            first = arg
    return first

def min3(*args):
    tmp = list(args)
    tmp.sort()
    return tmp[0]

def min4(*args):
    return sorted(args)[0]

for func in (min1, min2, min3, min4):           # Test all 4 functions
    print(func.__name__ + '...')
    print(func(3, 4.0, 1, 2))                   # Mixed numerics
    print(func('bb', 'aa'))                     # Strings: code points
    print(func([2, 2], [1, 1], [3, 3]))         # Lists: recursive
    print(func(*'hack'))                        # Unpacked characters

This script’s testing code uses the __name__ attribute we met earlier, along with a for loop to run each function one at a time (remember, functions are objects that work in a tuple too). All four solutions produce the same result when the file is run, so we’ll list just the first’s output here. Run this file live, or import it as a module and type a few calls to its function interactively to experiment with them on your own (see Chapter 3 for tips on both modes):

$ python3 mins.py
min1...
1
aa
[1, 1]
a
…and the same for others…

Notice that none of these four variants tests for the case where no arguments are passed in. They could, but there’s probably no point in doing so here—in all four solutions, Python will automatically raise an exception to signal the error if no arguments are sent. The first variant raises an exception when we try to fetch argument 0, the second when Python detects an argument list mismatch, and the third and fourth when we try to return item 0 post sort.

This is exactly what we want to happen—because these functions support any object (including None), there is no value that we could pass back to designate an error, so we may as well let the exception be raised. There are exceptions to this exceptions rule (e.g., you might test for errors yourself if you’d rather avoid actions that run before reaching the code that triggers an error automatically). But in general—and especially when errors aren’t common—it’s better to assume that arguments will work in your functions’ code, and let Python raise errors for you when they do not.

Bonus Points

You can get bonus points here for changing these functions to compute the arguments’ maximum, rather than minimum, value. This one’s trivial: the first two versions only require changing < to >, and the last two simply require that we return item [−1] instead of [0]. For an extra point, be sure to mod the function name to “max” as well (though this part is strictly optional).

True curve busters might also note that it’s possible to generalize a single function to compute either a minimum or a maximum value, by evaluating comparison expression strings with a tool like the eval built-in function (described in Python’s library manual, and at various appearances here, especially its note in Chapter 10), or by passing in an arbitrary comparison function. Example 18-3 shows how to implement the latter scheme for one of the coding options.

"Find minimum -or- maximum value of arguments"

def minmax(test, *args):
    res = args[0]
    for arg in args[1:]:
        if test(arg, res):
            res = arg
    return res

def lessthan(x, y): return x < y                # See also: lambda, eval
def grtrthan(x, y): return x > y

print(minmax(lessthan, 4, 2, 1, 5, 6, 3))       # Self-test code
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))

Running this script prints both minimum and maximum, per its self-test code at the end:

$ python3 minmax.py
1
6

Again, functions are just another kind of object, which allows them to be passed into other functions as done here. To make this a max (or other) function, for example, we simply pass in the right sort of test function to minmax. This may seem like extra work, but the main point of generalizing functions this way—instead of cutting and pasting to change just a single character—is that we’ll only have one version to change in the future, not two.

The Punch Line

Of course, all this was just a coding exercise. There’s really no reason to write min or max functions, because both are built-ins in Python! We met them briefly in Chapter 5 in conjunction with numeric tools, and again in Chapter 14 when exploring iteration contexts. The built-in versions work almost exactly like ours, but they’re coded in C for optimal speed and accept either a single iterable or multiple arguments. Still, though it’s superfluous in this context, the general coding pattern we used here might be useful in other scenarios.

Testing the Code

To test more thoroughly, the following continues our REPL session to code a function that applies the two tools to arguments in different orders using a simple shuffling technique that we saw in Chapter 13—it slices to move the first to the end on each loop, uses a * to unpack arguments, and sorts so results are comparable. Notice that arguments for the function are sent as a sequence (not discrete items), and the trace configuration option is keyword-only here:

>>> def tester(func, items, *, trace=True):
       for i in range(len(items)):
           items = items[1:] + items[:1]         # Move front item to back
           if trace: print(items)
           print(sorted(func(*items)))           # Test with reordered items

>>> tester(intersect, (s1, s2, s3))              # Use strings from prior listing
('CODE', 'CASH', 'HACKK')
['C']
('CASH', 'HACKK', 'CODE')
['C']
('HACKK', 'CODE', 'CASH')
['C']

>>> tester(union, (s1, s2, s3), trace=False)
['A', 'C', 'D', 'E', 'H', 'K', 'O', 'S']
['A', 'C', 'D', 'E', 'H', 'K', 'O', 'S']
['A', 'C', 'D', 'E', 'H', 'K', 'O', 'S']

>>> tester(intersect, (s1, s2, s3), trace=False)
['C']
['C']
['C']

Two context notes here: first, because duplicates won’t appear in these intersection and union functions, they qualify as set operations mathematically, but may not be optimal in term of speed. Still, there’s not much point in improving this demo’s code—intersection and union, like min and max, are built-in operations today: the set object we explored in Chapter 5 does intersection and union with & and |, and has methods that take multiple operands too. Hence, optimizing this code is left as suggested exercise, but see inter3.py in the examples package for pointers.

Second, the argument scrambling in the tester here doesn’t generate all possible argument orders (that would require a full permutation, and 6 orderings for 3 arguments), but suffices to check if argument order impacts results. As a preview, though, the tester would be simpler and more flexible if it delegated scrambling to a reusable function. Watch for this revision in Chapter 20, after we’ve explored how to code user-defined generators. We’ll also recode set tools one last time in Chapter 32 and a solution to a Part VI exercise, as classes that add them to the list object as methods.

Using Keyword-Only Arguments

Our print emulator works, but has a minor flaw baked in: it assumes that all positional arguments are to be printed, and all keywords are for options only. Any extra keyword arguments are silently ignored, and neither printed nor reported. A call like the following, for instance, will ignore the extra—and likely erroneous—sap argument:

$ python3
>>> from print3 import print3
>>> print3(3.12, 'py', sap='@')
3.12 py

It may not make sense to print superfluous keywords, but we can detect them manually by using dict.pop() to delete fetched keywords, and checking if the dictionary is not empty when we’re done. The version in Example 18-7 does—it’s equivalent to the original, but triggers a built-in exception with a raise statement when unexpected keyword arguments are sent by a call (this is partly preview: we’ll study exceptions and raise in depth in Part VII).

"Use keyword-collector arguments with deletion and defaults"
import sys

def print3(*args, **kargs):
    sep  = kargs.pop('sep', ' ')
    end  = kargs.pop('end', '\n')
    file = kargs.pop('file', sys.stdout)
    if kargs: raise TypeError(f'extra keywords: {kargs}')
    output = ''
    first  = True
    for arg in args:
        output += ('' if first else sep) + str(arg)
        first = False
    file.write(output + end)

Notice that this file’s name uses an underscore instead of a dash: because it’s to be imported, its name must follow the rules for variables of Chapter 11. This version works as before, but it now catches extraneous keyword arguments:

>>> from print3_pops import print3
>>> print3(3.12, 'py', sep='@')
3.12@py
>>> print3(3.12, 'py', sap='@')
TypeError: extra keywords: {'sap': '@'}

It’s OK to reimport the same print3 name from a different file here: this simply replaces the prior version just like reassigning any other variable (more on this later in this book). That being coded, this example could also use keyword-only arguments to automatically validate configuration arguments, as the final variant in Example 18-8 illustrates.

"Use keyword-only arguments to emulate print"
import sys

def print3(*args, sep=' ', end='\n', file=sys.stdout):
    output = ''
    first  = True
    for arg in args:
        output += ('' if first else sep) + str(arg)
        first = False
    file.write(output + end)

This version works the same way for valid calls, but catches invalid keywords with keyword-only arguments instead of manual code, and is a prime example of how keyword-only arguments can address coding needs:

>>> from print3_kwonly import print3
>>> print3(3.12, 'py', sep='@')
3.12@py
>>> print3(3.12, 'py', sap='@')
TypeError: print3() got an unexpected keyword argument 'sap'

Given that this version of the function also requires four fewer lines of code than its predecessor, keyword-only arguments can simplify a specific category of functions that accept both arguments and options. A similar case can be made for positional-inly arguments versus manual code, but it’s more obscure, and this chapter has run out of space. For another example of keyword-only arguments at work, stay tuned for the iteration-timing case study in Chapter 21.

And for more inspiration, also see the sidebar “Why You Will Care: Customizing open” at the end of the previous chapter. Much as we did there, our print emulator could be assigned to builtins.print to replace the built-in with our custom version everywhere in a program. There’s no reason to do that for a version that’s the same, of course, but this technique can be used to install a replacement printer that mods or extends the built-in (e.g., with logging).