Motivation

Implementing instance Configuration in Code is easier than externally managing parameter values, as per the other patterns in this chapter, if you don’t have the necessary tools and support mechanisms. It can also be appealing to have the configuration with the code, where it is all visible in one place.

Consequences

Combining instance-specific configuration with code that is reused across the instances couples the code tightly to the environments it’s used in. A configuration change requires changing stack code, which needs more comprehensive testing than a change to an input value.

Creating a new environment involves making changes to all the stack code that includes configuration. The overhead of changing and testing stack code increases the cost of creating environments, which can lead to constraints on the capacity for delivery and testing software and infrastructure.

Infrastructure code (or any code) that includes instance-specific variations can be difficult to read, understand, and maintain as it grows. The preceding example code shows one part of a stack with conditional code based on the instance. The rest of the stack may have multiple other settings and options that also depend on which instance it’s applied to.

Implementation

Instance-specific code may be cleaned up by separating code that sets the values from the code where they are used. Here’s an example, changed from the preceding one:

if ${environment} == "dev"
  ${min_workers} = 1
  ${max_workers} = 1
elsif ${environment} == "test"
  ${min_workers} = 1
  ${max_workers} = 2
elsif ${environment} == "prod"
  ${min_workers} = 2
  ${max_workers} = 5
end

container_cluster: web_cluster-${environment}
  min_workers: ${min_workers}
  max_workers: ${max_workers}

Configuration and implementation are separated, which makes the code easier to read and maintain. But configuration changes still require changing the stack code. You need to either fully test the stack code for every configuration change or rely on humans to decide when a change needs testing. “I didn’t change anything important” is a phrase often heard during postmortems after production incidents.

This cleaned-up code example is only a step away from the Stack Configuration Files pattern described later in this chapter, which avoids most of the problems with the Configuration in Code antipattern.

Rather than specifying variations in infrastructure code based on the instance, specify the variations based on the specific outcome for each variation. You can then select the variation by using a parameter value that makes its purpose explicit.

For example, the following code configures our container cluster based on a parameter called cluster_size_, which can be set to S, M, or L:

if ${cluster_size} == "S"
  ${min_workers} = 1
  ${max_workers} = 1
elsif ${cluster_size} == "M"
  ${min_workers} = 1
  ${max_workers} = 2
elsif ${cluster_size} == "L"
  ${min_workers} = 2
  ${max_workers} = 5
end

container_cluster: web_cluster-${environment}
  min_workers: ${min_workers}
  max_workers: ${max_workers}

Changes to the stack code can be run through tests for all these size settings. The size of an instance can be changed without needing to change or test the stack code since the behavior is known.

Related patterns

As mentioned in the preceding section, the Stack Configuration Files pattern can be an easy step in moving away from this antipattern, by extracting instance-specific configuration out of the stack code.

Motivation

It’s dirt-simple to type values on the command line, which is helpful when you’re learning how to use a tool. Typing parameters on the command line is also useful for experimenting. Some teams write documents with the commands to run so that people can copy and paste them into a command prompt as needed.

Consequences

Making a mistake is easy when typing a value on the command line. Remembering which values to type can also be hard. For infrastructure that people care about, you probably don’t want to risk accidentally breaking something important by mistyping a command when making an improvement or fix. When multiple people work on an infrastructure stack, as in a team, you can’t expect everyone to remember the correct values to type for each instance. Even copying and pasting commands from a runbook can lead to errors.³

Implementation

For the example stack parameters, pass the values on the command line according to the syntax expected by the particular tool. With my fictional stack tool, the command looks like this:

stack up \
    environment=test \
    min_workers=1 \
    max_workers=1 \
    ssl_cert_passphrase="correct horse battery staple"

Anyone who runs the command needs to know the secrets, like passwords and keys, to use for a given environment and pass them on the command line. Your team should use a team password management tool to securely store and share these secrets among team members, and rotate secrets when people leave the team.⁴

Related patterns

The Scripted Parameters pattern takes the command that you would type and puts it into a script. The Pipeline Stack Parameters pattern does the same thing but puts the command and the parameter values into the pipeline configuration instead of a script.

Motivation

Environment variables can be useful when a relationship exists between where the infrastructure tool is executed and a particular environment instance. A common use case is for infrastructure developers to preset environment variables for their personal IaaS environment (see “Personal IaaS Environments”). Setting these values in a login script or shell configuration allows the developer to run infrastructure tool commands without having to think about which parameter values to use.

Consequences

Using environment variables directly in stack code arguably couples stack code too tightly to the runtime environment. Often, environment variables are appropriate for some environments, like developer workstations, but not in others, like pipeline stages, so you may need to implement multiple configuration patterns for the same stack. Setting secrets in environment variables may expose them to other processes that run on the same system.

Implementation

With many infrastructure coding languages, you can directly reference environment variables, as in the preceding example. However, it’s more common to use a wrapper script that runs the infrastructure tool, and have that script import environment variables. A wrapper script might implement alternative patterns for setting parameter values, using environment variables if they are set, but using a different approach if not.

Because infrastructure developers often work with multiple projects, they may find it useful to set different environment variables for different projects. Local development solutions like those described in “Working on Code Locally” can create separate environments for each project, also including different tools and versions. Lighter solutions focus on setting environment variables, such as direnv, which loads directory-specific environment settings.

Motivation

Scripts are a simple way to capture the values for each instance, avoiding the problems with the Manual Stack Parameters antipattern. You can be confident that values are used consistently for each environment. By checking the script into version control, you ensure that you are tracking any changes to the configuration values.

Applicability

A stack provisioning script is a useful way to set parameters when you have a fixed set of environments that don’t change very often. The script doesn’t require the additional moving parts of some of the other patterns in this chapter.

Because hardcoding secrets in scripts is dangerous, this pattern is not suitable for secrets. That doesn’t mean you shouldn’t use this pattern, only that you’ll need to implement a separate pattern for dealing with secrets (see “Handling Secrets” for suggestions).

Consequences

Commands used to run the stack tool become complicated over time. Infrastructure deployment orchestration scripts can grow into messy beasts. “Infrastructure Deployment Scripts” discusses scripts for running deployments, and “Using Delivery Orchestration Scripts” describes scripting the overall infrastructure delivery process.

Implementation

This pattern has two common implementations. One is a single script that takes the environment as a command-line argument, with hardcoded parameter values for each environment:

#!/bin/sh

case $1 in
dev)
  MIN_WORKERS=1
  MAX_WORKERS=1
  ;;
test)
  MIN_WORKERS=2
  MAX_WORKERS=3
  ;;
prod)
  MIN_WORKERS=2
  MAX_WORKERS=6
  ;;
*)
  echo "Unknown environment $1"
  exit 1
  ;;
esac

stack up \
    environment=$1 \
    min_workers=${MIN_WORKERS} \
    max_workers=${MAX_WORKERS}

Another implementation is a separate script for each stack instance:

our-infra-stack/
  ├── bin/
  │   ├── dev.sh
  │   ├── test.sh
  │   └── prod.sh
  ├── src/
  └── test/

Each of these scripts is identical but has different parameter values hardcoded in it. The scripts are smaller because they don’t need logic to select parameter values. However, the scripts need more maintenance. If you need to change the command, you need to change it across all the scripts. Having a script for each environment also tempts people to overly customize environments, creating inconsistency.

Commit your provisioning script or scripts to source control. Putting your script in the same project as the stack it provisions ensures that it stays in sync with the stack code. For example, if you add a new parameter, you add it to the infrastructure source code as well as to your provisioning script. You always know which version of the script to run for a given version of the stack code.

As mentioned earlier, you shouldn’t hardcode secrets into scripts, so you’ll need to use a different pattern for those. You can use the script to support that pattern. In this example, a command-line tool fetches the secret from a secrets manager, following the Stack Parameter Registry pattern:

...
# (Set environment specific values as in other examples)
...

SSL_CERT_PASSPHRASE=$(secrets-tool get-secret id="/ssl_cert_passphrase/${ENV}")

stack up \
    environment=${ENV} \
    min_workers=${MIN_WORKERS} \
    max_workers=${MAX_WORKERS} \
    ssl_cert_passphrase="${SSL_CERT_PASSPHRASE}"

The secrets-tool command connects to the secrets manager and retrieves the secret for the relevant environment by using the ID /ssl_cert_passphrase/${ENV}. This example assumes that the session is authorized to use the secrets manager. An infrastructure developer may use the tool to start a session before running this script. Or the compute instance that runs the script may have secrets injected into its environment, as discussed later.

Motivation

Creating configuration files for a stack’s instances is straightforward. Because the file is committed to the source code repository, it is easy to do the following:

• See which values are used for any given environment (“What is the maximum cluster size for production?”)

• Trace the history for debugging (“When did the maximum cluster size change?”)

• Audit changes (“Who changed the maximum cluster size?”)

The Stack Configuration Files pattern enforces the separation of configuration from the stack code with few moving parts.

Applicability

Stack configuration files are appropriate when the number of environments doesn’t change often. You must add a file to your project in order to add a new stack instance. The files also require (and help enforce) consistent logic in the way different instances are created, since, unlike scripts, the configuration files can’t include logic.

Consequences

When you want to create a new stack instance, you need to add a new configuration file to the stack project. Doing this means you can’t create new environments on the fly, programmatically. In “Ephemeral Test Stack”, I describe an approach for managing test environments that relies on creating environments automatically. You could work around this by automatically generating a configuration file.

Parameter files can add friction when changing the configuration of downstream environments in a change delivery pipeline of the kind described in “Designing Infrastructure Delivery Pipelines”. Every change to the stack project code must progress through each stage of the pipeline before being applied to production. Completion can take a while, and this process doesn’t add any value when the configuration change is applied only to production.

Defining parameter values can be a source of considerable complexity in deployment scripts, and I’ll talk about this more in “Using Delivery Orchestration Scripts”. As a teaser, consider that teams often want to define default values for stack projects and for environments, and then need logic to combine these into values for a given instance of a given stack in a different environment. Inheritance models for parameter values can get messy and confusing.

Configuration files in source control should not include secrets. So to handle secrets, you either need to select an additional pattern from this chapter or implement a separate secrets configuration file outside of source control.

Implementation

You can define stack parameter values in a separate file for each environment, as shown in Figure 8-2.

The contents of a parameter file could look like this:

environment = test
min_workers = 1
max_workers = 3

Pass the path to the relevant parameter file when running the stack command:

stack up --source ./src --config ./environments/test.properties

If the system is composed of multiple stacks, managing configuration across all the environments can get messy. There are two common ways of arranging parameter files in these cases. One is to put configuration files for all the environments with the code for each stack:

   ├── cluster_stack/
   │   ├── src/
   │   │   ├── cluster.infra
   │   │   ├── host_servers.infra
   │   │   └── networking.infra
   │   └── environments/
   │       ├── dev.properties
   │       ├── test.properties
   │       └── prod.properties
   │
   └── appserver_stack/
       ├── src/
       │   ├── server.infra
       │   └── networking.infra
       └── environments/
           ├── dev.properties
           ├── test.properties
           └── prod.properties

The other is to centralize the configuration for all the stacks in one place:

   ├── cluster_stack/
   │   ├── cluster.infra
   │   ├── host_servers.infra
   │   └── networking.infra
   │
   ├── appserver_stack/
   │   ├── server.infra
   │   └── networking.infra
   │
   └── environments/
       ├── dev/
       │   ├── cluster.properties
       │   └── appserver.properties
       ├── test/
       │   ├── cluster.properties
       │   └── appserver.properties
       └── production/
           ├── .properties
           └── appserver.properties

Each approach can become messy and confusing in different ways. When you need to make a change to all the elements in an environment, making changes to configuration files across dozens of stack projects is painful. When you need to change the configuration for a single stack across the various environments it’s in, trawling through a tree full of configurations for dozens of other stacks is also not fun.

Most infrastructure orchestration tools, like Terragrunt, define folder and file structures for configuration files. If you use configuration files to provide secrets, rather than using a separate pattern for secrets, you should manage those files outside of the project code checked into source control.

For local development environments, you can require users to manually create the file in a set location. Pass the file location to the stack command like this:

stack up --source ./src \
  --config ./environments/test.properties \
  --config ../.secrets/test.properties

In this example, you provide two --config arguments to the stack tool, and it reads parameter values from both. You have a directory named .secrets outside the project folder, so it is not in source control.

Doing this can be trickier when running the stack tool automatically from a compute instance such as a CD pipeline agent. You could provision similar secrets property files onto these compute instances, but that can expose secrets to other processes that run on the same agent. You also need to provide the secrets to the process that builds the compute instance for the agent, so you still have a bootstrapping problem.

Related patterns

Putting configuration values into files simplifies the approach of provisioning scripts or wrapper scripts. You can avoid some of the limitations of environment configuration files by using the Stack Parameter Registry pattern instead. A simple implementation of the parameter registry pattern moves the parameter files for stacks out of each stack project and into a central location, such as an infrastructure-configuration Git repository.

Motivation

If you’re using a pipeline tool to run your infrastructure deployment tool, the pipeline tool provides the mechanism for storing and passing parameter values to the deployment tool out of the box. Assuming your pipeline tool is itself configured by code, the values are defined as code and stored in version control.

Configuration values are kept separate from the infrastructure code. You can change configuration values for downstream environments and apply them immediately, without needing to push through a new version of the infrastructure code from the start of the pipeline.

Applicability

Teams that are already using a pipeline to apply infrastructure code to environments can easily leverage this pattern to set stack parameters for each instance. However, if stacks require more than a few parameter values, setting these in the pipeline configuration has serious drawbacks, so you should avoid this.

Consequences

Setting stack instance variables in the pipeline configuration couples configuration values to your delivery implementation. The more configuration values you define in your pipeline, the harder it is to run the stack tool outside the pipeline. Your pipeline can become a single point of failure—you may not be able to fix, recover, or rebuild an environment in an emergency until you have recovered your pipeline. And it can be hard for your team to develop and test stack code outside the pipeline.

In general, it’s best to keep the pipeline configuration for applying a stack project as small and simple as possible. Most of the logic should live in a script called by the pipeline, rather than in the pipeline configuration.

Implementation

Parameters should be implemented using “as code” configuration of the pipeline tool. The following example shows a pseudocode pipeline stage configuration that passes the values on the command line:

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src environment=test min_workers=1 max_workers=1
    stack test environment=test

You can also set parameters as environment variables that the stack code uses:

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  environment_vars:
    STACK_ENVIRONMENT=test
    STACK_MIN_WORKERS=1
    STACK_MAX_WORKERS=1
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src
    stack test environment=test

In this example, the pipeline tool sets those environment variables before running the commands.

Many pipeline tools provide management features that you can use to pass secrets to your stack command. Set the secret values in the pipeline tool’s configuration and then refer to them in your pipeline job:

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src environment=test \
        min_workers=1 \
        max_workers=1 \
        ssl_cert_passphrase=${STACK_SSL_CERT_PASSPHRASE}

Note that pipeline tools tend to log output from their jobs, and this output may include secrets set this way. See “Handling Secrets” for better approaches.

Related patterns

Defining the commands and parameters to apply stack code for each environment in a pipeline configuration is similar to the Scripted Parameters pattern. The difference is where the scripting lives—in the pipeline configuration versus in script files.

Motivation

Storing parameter values in a registry separates configuration from implementation. Parameters in a registry can be set, used, and viewed by multiple tools, using various languages and technologies. This flexibility reduces coupling between parts of the system. You can replace any tool that uses the registry without affecting any other tool that uses it.

Because they are tool-agnostic, stack parameter registries can act as a source of truth for infrastructure and even system configuration, acting as a configuration management database (CMDB). This configuration data can be useful in regulated contexts, making it easy to generate reports for auditing.

Applicability

If you are using a configuration registry for other purposes, using it for stack parameters as well may make sense. For example, a configuration registry can be used to integrate multiple stacks (see “Integration Registry Lookup”).

Consequences

A stack parameter registry requires a configuration registry service, which is an extra moving part for your overall system. The registry is a dependency for your stack and a potential point of failure. If the registry becomes unavailable, reprovisioning or updating your infrastructure may be impossible until you can restore it. This dependency can be painful in disaster-recovery scenarios, putting the registry service on the critical path.

Managing parameter values separately from the stack code that uses it has trade-offs. You can change the configuration of a stack instance without making a change to the stack project. If one team maintains a reusable stack project, other teams can use it to create their own stack instances without needing to add or change configuration files in the stack project itself, as they may with other patterns like Stack Configuration Files. On the other hand, making changes across more than one place—stack project and parameter registry—adds complexity and opportunity for mistakes.

Implementation

I’ll discuss ways to implement a parameter registry in “Implementing a Configuration Registry”. In short, it may be a service that stores key-value pairs, or it could be a file or directory structure that contains key-value pairs. Either way, parameter values can usually be stored in a hierarchical structure, so you can store and find them based on the environment and the stack, and perhaps other factors like the application, service, team, geography, or customer.

The values for this chapter’s example container cluster could look like Example 8-2.

└── env/
    ├── dev/
    │   └── cluster/
    │        ├── min = 1
    │        └── max = 1
    ├── test/
    │   └── cluster/
    │        ├── min = 2
    │        └── max = 3
    └── prod/
        └── cluster/
             ├── min = 2
             └── max = 6

When you apply the infrastructure stack code to an instance, the stack tool uses the key to retrieve the relevant value. You pass the environment parameter to the stack tool, and the code uses this to refer to the relevant location in the registry:

cluster:
  id: container_cluster-${environment}
  minimum: ${get_value("/env/${environment}/cluster/min")}
  maximum: ${get_value("/env/${environment}/cluster/max")}

The get_registry_item() function in the stack code looks up the value.

This implementation ties your stack code to the configuration registry. You need the registry to run and test your code, which can be too heavy. You could work around this by fetching the values from the registry in a script, which then passes them to the stack code as normal parameters. Doing this gives you the flexibility to set parameter values in other ways. This is particularly useful for reusable stack code, giving users of your code more options for configuring their stack instances.

Secrets management services (see “Storage resources”) are a special type of parameter registry. Used correctly, they ensure that secrets are available only to people and services that require them, without exposing them more widely. Some configuration registry products and services can be used to store both secret and nonsecret values. But it’s important to avoid storing secrets in registries that don’t protect them. Doing so makes the registry an easy target for attackers.

Related patterns

You probably need to pass at least one parameter to the stack tool to indicate which stack instance’s parameters to use. You might use either the Scripted Parameters or the Pipeline Stack Parameter pattern for this.