Motivation

Most platforms and tools support environment variables, so it’s easy to do.

Applicability

If you’re already using environment variables in your system and have suitable mechanisms to manage them, you might find it convenient to use them for stack parameters.

Consequences

You need to use an additional pattern from this chapter to get the values to set. Doing this adds moving parts, making it hard to trace configuration values for any given stack instance, and more work to change the values.

Using environment variables directly in stack code, as in Example 7-5, arguably couples stack code too tightly to the runtime environment.

Setting secrets in environment variables may expose them to other processes that run on the same system.

Implementation

Again, you need to set the environment variables to use, which means selecting another pattern from this chapter. For example, if you expect people to set environment variables in their local environment to apply stack code, you are using the manual stack parameters antipattern (see “Antipattern: Manual Stack Parameters”). You could set them in a script that runs the stack tool (the scripted parameters pattern, as discussed in “Pattern: Scripted Parameters”), or have the pipeline tool set them (see “Pattern: Pipeline Stack Parameters”).

Another approach is to put the values into a script that people or instances import into their local environment. This is a variation of the stack configuration files pattern (see “Pattern: Stack Configuration Files”). The script to set the variables would be exactly like Example 7-4, and any command that runs the stack tool would import it into the environment:

source ./environments/staging.env
stack up --source ./src

Alternatively, you could build the environment values into a compute instance that runs the stack tool. For example, if you provision a separate CD agent node to run the stack tool to build and update stacks in each environment, the code to build the node could set the appropriate values as environment variables. Those environment variables would be available to any command that runs on the node, including the stack tool.

But to do this, you need to pass the values to the code that builds your agent nodes. So you need to select another pattern from this chapter to do that.

The other side of implementing this pattern is how the stack tool gets the environment values. Example 7-5 showed how stack code can directly read environment variables.

But you could, instead, use a stack orchestration script (see “Using Scripts to Wrap Infrastructure Tools”) to read the environment variables and pass them to the stack tool on the command line. The code in the orchestration script would look like this:

stack up \
  environment=${STACK_ENVIRONMENT} \
  cluster_minimum=${STACK_CLUSTER_MINIMUM} \
  cluster_maximum=${STACK_CLUSTER_MAXIMUM} \
  ssl_cert_passphrase="${STACK_SSL_CERT_PASSPHRASE}"

This approach decouples your stack code from the environment it runs in.

Related patterns

Any of the other patterns in this chapter can be combined with this one to set environment values.

Scripts are a simple way to capture the values for each instance, avoiding the problems with the manual stack parameters antipattern (see “Antipattern: Manual Stack Parameters”). You can be confident that values are used consistently for each environment. By checking the script into version control, you ensure you are tracking any changes to the configuration values.

Applicability

A stack provisioning script is a useful way to set parameters when you have a fixed set of environments that don’t change very often. It doesn’t require the additional moving parts of some of the other patterns in this chapter.

Because it is wrong to hardcode secrets in scripts, this pattern is not suitable for secrets. That doesn’t mean you shouldn’t use this pattern, only that you’ll need to implement a separate pattern for dealing with secrets (see “Handling Secrets as Parameters” for suggestions).

Consequences

It’s common for the commands used to run the stack tool to become complicated over time. Provisioning scripts can grow into messy beasts. “Using Scripts to Wrap Infrastructure Tools” discusses how these scripts are used and outlines pitfalls and recommendations for keeping them maintainable. You should test provisioning scripts since they can be a source of issues with the systems they provision.

Implementation

There are two common implementations for this pattern. One is a single script that takes the environment as a command-line argument, with hardcoded parameter values for each environment. Example 7-6 is a simple example of this.

#!/bin/sh

case $1 in
test)
  CLUSTER_MINIMUM=1
  CLUSTER_MAXIMUM=1
  ;;
staging)
  CLUSTER_MINIMUM=2
  CLUSTER_MAXIMUM=3
  ;;
production)
  CLUSTER_MINIMUM=2
  CLUSTER_MAXIMUM=6
  ;;
*)
  echo "Unknown environment $1"
  exit 1
  ;;
esac

stack up \
    environment=$1 \
    cluster_minimum=${CLUSTER_MINIMUM} \
    cluster_maximum=${CLUSTER_MAXIMUM}

Another implementation is a separate script for each stack instance, as shown in Example 7-7.

our-infra-stack/
  ├── bin/
  │   ├── test.sh
  │   ├── staging.sh
  │   └── production.sh
  ├── src/
  └── test/

Each of these scripts is identical but has different parameter values hardcoded in it. The scripts are smaller because they don’t need logic to select between different parameter values. However, they need more maintenance. If you need to change the command, you need to make it across all of the scripts. Having a script for each environment also tempts people to customize different environments, creating inconsistency.

Commit your provisioning script or scripts to source control. Putting it in the same project as the stack it provisions ensures that it stays in sync with the stack code. For example, if you add a new parameter, you add it to the infrastructure source code and also to your provisioning script. You always know which version of the script to run for a given version of the stack code.

“Using Scripts to Wrap Infrastructure Tools” discusses the use of scripts to run stack tools in much more detail.

As mentioned earlier, you shouldn’t hardcode secrets into scripts, so you’ll need to use a different pattern for those. You can use the script to support that pattern. In Example 7-8, a command-line tool fetches the secret from a secrets manager, following the parameter registry pattern (see “Pattern: Stack Parameter Registry”).

...
# (Set environment specific values as in other examples)
...

SSL_CERT_PASSPHRASE=$(some-tool get-secret id="/ssl_cert_passphrase/${ENV}")

stack up \
    environment=${ENV} \
    cluster_minimum=${CLUSTER_MINIMUM} \
    cluster_maximum=${CLUSTER_MAXIMUM} \
    ssl_cert_passphrase="${SSL_CERT_PASSPHRASE}"

The some-tool command connects to the secrets manager and retrieves the secret for the relevant environment using the ID /ssl_cert_passphrase/${ENV}. This example assumes the session is authorized to use the secrets manager. An infrastructure developer may use the tool to start a session before running this script. Or the compute instance that runs the script may be authorized to retrieve secrets using secretless authorization (as I describe in “Secretless Authorization”).

Related patterns

Provisioning scripts run the command-line tool for you, so are a way to move beyond the manual stack parameters antipattern (see “Antipattern: Manual Stack Parameters”). The stack configuration files pattern extracts the parameter values from the script into separate files.

Motivation

Creating configuration files for a stack’s instances is straightforward and easy to understand. Because the file is committed to the source code repository, it is easy to:

• See what values are used for any given environment (“What is the maximum cluster size for production?”)

• Trace the history for debugging (“When did the maximum cluster size change?”)

• Audit changes (“Who changed the maximum cluster size?”)

Stack configuration files enforce the separation of configuration from the stack code.

Applicability

Stack configuration files are appropriate when the number of environments doesn’t change often. They require you to add a file to your project to add a new stack instance. They also require (and help enforce) consistent logic in how different instances are created and updated, since the configuration files can’t include logic.

Consequences

When you want to create a new stack instance, you need to add a new configuration file to the stack project. Doing this prevents you from automatically creating new environments on the fly. In “Pattern: Ephemeral Test Stack”, I describe an approach for managing test environments that relies on creating environments automatically. You could work around this by creating a configuration file for an ephemeral environment on demand.

Parameter files can add friction for changing the configuration of downstream environments in a change delivery pipeline of the kind described in “Infrastructure Delivery Pipelines”. Every change to the stack project code must progress through each stage of the pipeline before being applied to production. It can take a while for this to complete and doesn’t add any value when the configuration change is only applied to production.

Defining parameter values can be a source of considerable complexity in provisioning scripts. I’ll talk about this more in “Using Scripts to Wrap Infrastructure Tools”, but as a teaser, consider that teams often want to define default values for stack projects, and for environments, and then need logic to combine these into values for a given instance of a given stack in a different environment. Inheritance models for parameter values can get messy and confusing.

Configuration files in source control should not include secrets. So for secrets, you either need to select an additional pattern from this chapter to handle secrets or implement a separate secrets configuration file outside of source control.

You define stack parameter values in a separate file for each environment, as shown earlier in Example 7-9.

The contents of a parameter file could look like this:

env  = staging
cluster_minimum = 2
cluster_maximum = 3

Pass the path to the relevant parameter file when running the stack command:

stack up --source ./src --config ./environments/staging.properties

If the system is composed of multiple stacks, then it can get messy to manage configuration across all of the environments. There are two common ways of arranging parameter files in these cases. One is to put configuration files for all of the environments with the code for each stack:

   ├── cluster_stack/
   │   ├── src/
   │   │   ├── cluster.infra
   │   │   ├── host_servers.infra
   │   │   └── networking.infra
   │   └── environments/
   │       ├── test.properties
   │       ├── staging.properties
   │       └── production.properties
   │
   └── appserver_stack/
       ├── src/
       │   ├── server.infra
       │   └── networking.infra
       └── environments/
           ├── test.properties
           ├── staging.properties
           └── production.properties

The other is to centralize the configuration for all of the stacks in one place:

   ├── cluster_stack/
   │   ├── cluster.infra
   │   ├── host_servers.infra
   │   └── networking.infra
   │
   ├── appserver_stack/
   │   ├── server.infra
   │   └── networking.infra
   │
   └── environments/
       ├── test/
       │   ├── cluster.properties
       │   └── appserver.properties
       ├── staging/
       │   ├── cluster.properties
       │   └── appserver.properties
       └── production/
           ├── cluster.properties
           └── appserver.properties

Each approach can become messy and confusing in its own way. When you need to make a change to all of the things in an environment, making changes to configuration files across dozens of stack projects is painful. When you need to change the configuration for a single stack across the various environments it’s in, trawling through a tree full of configuration for dozens of other stacks is also not fun.

If you want to use configuration files to provide secrets, rather than using a separate pattern for secrets, then you should manage those files outside of the project code checked into source control.

For local development environments, you can require users to create the file in a set location manually. Pass the file location to the stack command like this:

stack up --source ./src \
  --config ./environments/staging.properties \
  --config ../.secrets/staging.properties

In this example, you provide two --config arguments to the stack tool, and it reads parameter values from both. You have a directory named .secrets outside the project folder, so it is not in source control.

It can be trickier to do this when running the stack tool automatically, from a compute instance like a CD pipeline agent. You could provision similar secrets property files onto these compute instances, but that can expose secrets to other processes that run on the same agent. You also need to provide the secrets to the process that builds the compute instance for the agent, so you still have a bootstrapping problem.

Related patterns

Putting configuration values into files simplifies the provisioning scripts described in “Pattern: Scripted Parameters”. You can avoid some of the limitations of environment configuration files by using the stack parameter registry pattern instead (see “Pattern: Stack Parameter Registry”). Doing this moves parameter values out of the stack project code and into a central location, which allows you to use different workflows for code and configuration.

Motivation

A wrapper stack leverages the stack tool’s module functionality or library support to reuse shared code across stack instances. You can use the tool’s module versioning, dependency management, and artifact repository functionality to implement a change delivery pipeline (see “Infrastructure Delivery Pipelines”). As of this writing, most infrastructure stack tools don’t have a project packaging format that you can use to implement pipelines for stack code. So you need to create a custom stack packaging process yourself. You can work around this by using a wrapper stack, and versioning and promoting your stack code as a module.

With wrapper stacks, you can write the logic for provisioning and configuring stacks in the same language that you use to define your infrastructure, rather than using a separate language as you would with a provisioning script (see “Pattern: Scripted Parameters”).

Consequences

Components add an extra layer of complexity between your stack and the code contained in the component. You now have two levels: the stack project, which contains the wrapper projects, and the component that contains the code for the stack.

Because you have a separate code project for each stack instance, people may be tempted to add custom logic for each instance. Custom instance code makes your codebase inconsistent and hard to maintain.

Each stack instance has a separate infrastructure stack project. For example, you would have a separate Terraform project for each environment. You can implement this like a copy-paste environment (see “Antipattern: Copy-Paste Environments”), with each environment in a separate repository.

Alternatively, each environment project could be a folder in a single repository:

my_stack/
   ├── test/
   │   └── stack.infra
   ├── staging/
   │   └── stack.infra
   └── production/
       └── stack.infra

Define the infrastructure code for the stack as a module, according to your tool’s implementation. You could put the module code in the same repository with your wrapper stacks. However, this would prevent you from leveraging module versioning functionality. That is, you wouldn’t be able to use different versions of the infrastructure code in different environments, which is crucial for progressively testing your code.

The following example is a wrapper stack that imports a module called container_cluster_module, specifying the version of the module, and the configuration parameters to pass to it:

module:
  name: container_cluster_module
  version: 1.23
  parameters:
    env: test
    cluster_minimum: 1
    cluster_maximum: 1

The wrapper stack code for the staging and production environments is similar, other than the parameter values, and perhaps the module version they use.

The project structure for the module could look like this:

   ├── container_cluster_module/
   │   ├── cluster.infra
   │   └── networking.infra
   └── test/

When you make a change to the module code, you test and upload it to a module repository. How the repository works depends on your particular infrastructure stack tool. You can then update your test stack instance to import the new module version and apply it to the test environment.

Terragrunt is a stack orchestration tool that implements the wrapper stack pattern.

Related patterns

A wrapper stack is similar to the scripted parameters pattern. The main differences are that it uses your stack tool’s language rather than a separate scripting language and that the infrastructure code is in a separate component.

Motivation

If you’re using a pipeline tool to run your infrastructure stack tool, it provides the mechanism for storing and passing parameter values to the tool out of the box. Assuming your pipeline tool is itself configured by code, then the values are defined as code and stored in version control.

Configuration values are kept separate from the infrastructure code. You can change configuration values for downstream environments and apply them immediately, without needing to progress a new version of the infrastructure code from the start of the pipeline.

Applicability

Teams who are already using a pipeline to apply infrastructure code to environments can easily leverage this to set stack parameters for each environment. However, if stacks require more than a few parameter values, defining these in the pipeline configuration has serious drawbacks, so you should avoid this.

Consequences

By defining stack instance variables in the pipeline configuration, you couple configuration values with your delivery process. There is a risk of the pipeline configuration becoming complicated and hard to maintain.

The more configuration values you define in your pipeline, the harder it is to run the stack tool outside the pipeline. Your pipeline can become a single point of failure—you may not be able to fix, recover, or rebuild an environment in an emergency until you have recovered your pipeline. And it can be hard for your team to develop and test stack code outside the pipeline.

In general, it’s best to keep the pipeline configuration for applying a stack project as small and simple as possible. Most of the logic should live in a script called by the pipeline, rather than in the pipeline configuration.

Parameters should be implemented using “as code” configuration of the pipeline tool. Example 7-10 shows a pseudocode pipeline stage configuration.

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src environment=test cluster_minimum=1 cluster_maximum=1
    stack test environment=test

This example passes the values on the command line. You may also set them as environment variables that the stack code uses, as shown in Example 7-11 (see also “Pattern: Stack Environment Variables”).

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  environment_vars:
    STACK_ENVIRONMENT=test
    STACK_CLUSTER_MINIMUM=1
    STACK_CLUSTER_MAXIMUM=1
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src
    stack test environment=test

In this example, the pipeline tool sets those environment variables before running the commands.

Many pipeline tools provide secret management features that you can use to pass secrets to your stack command. You set the secret values in the pipeline tool in some fashion, and can then refer to them in your pipeline job, as shown in Example 7-12.

stage: apply-test-stack
  input_artifacts: container_cluster_stack
  commands:
    unpack ${input_artifacts}
    stack up  --source ./src environment=test \
        cluster_minimum=1 \
        cluster_maximum=1 \
        ssl_cert_passphrase=${STACK_SSL_CERT_PASSPHRASE}

Related patterns

Defining the commands and parameters to apply stack code for each environment in pipeline configuration is similar to the scripted parameters pattern. The difference is where the scripting lives—in the pipeline configuration versus in script files.

Motivation

Storing parameter values in a registry separates configuration from implementation. Parameters in a registry can be set, used, and viewed by different tools, using different languages and technologies. This flexibility reduces coupling between different parts of the system. You can replace any tool that uses the registry without affecting any other tool that uses it.

Because they are tool-agnostic, stack parameter registries can act as a source of truth for infrastructure and even system configuration, acting as a Configuration Management Database (CMDB). This configuration data can be useful in regulated contexts, making it easy to generate reports for auditing.

If you are using a configuration registry for other purposes, it makes sense to use it as a stack parameter registry, as well. For example, a configuration registry is a useful way to integrate multiple stacks (see “Discovering Dependencies Across Stacks”).

Consequences

A stack parameter registry requires a configuration registry, which is an extra moving part for your overall system. The registry is a dependency for your stack and a potential point of failure. If the registry becomes unavailable, it may be impossible to re-provision or update the infrastructure stack until you can restore it. This dependency can be painful in disaster recovery scenarios, putting the registry service on the critical path.

Managing parameter values separately from the stack code that uses it has trade-offs. You can change the configuration of a stack instance without making a change to the stack project. If one team maintains a reusable stack project, other teams can use it to create their own stack instances without needing to add or change configuration files in the stack project itself.

On the other hand, making changes across more than one place—stack project and parameter registry—adds complexity and opportunity for mistakes.

Implementation

I’ll discuss ways to implement a parameter registry in “Configuration Registry”. In short, it may be a service that stores key/value pairs, or it could be a file or directory structure of files that contain key-value pairs. Either way, parameter values can usually be stored in a hierarchical structure, so you can store and find them based on the environment and the stack, and perhaps other factors like the application, service, team, geography, or customer.

The values for this chapter’s example container cluster could look like Example 7-13.

└── env/
    ├── test/
    │   └── cluster/
    │        ├── min = 1
    │        └── max = 1
    ├── staging/
    │   └── cluster/
    │        ├── min = 2
    │        └── max = 3
    └── production/
        └── cluster/
             ├── min = 2
             └── max = 6

When you apply the infrastructure stack code to an instance, the stack tool uses the key to retrieve the relevant value. You will need to pass the environment parameter to the stack tool, and the code uses this to refer to the relevant location in the registry:

cluster:
  id: container_cluster-${environment}
  minimum: ${get_value("/env/${environment}/cluster/min")}
  maximum: ${get_value("/env/${environment}/cluster/max")}

The get_registry_item() function in the stack code looks up the value.

This implementation ties your stack code to the configuration registry. You need the registry to run and test your code, which can be too heavy. You could work around this by fetching the values from the registry in a script, which then passes them to the stack code as normal parameters. Doing this gives you the flexibility to set parameter values in other ways. For reusable stack code this is particularly useful, giving users of your code more options for how to configure their stack instances.

Secrets management services (see “Storage Resources”) are a special type of parameter registry. Used correctly, they ensure that secrets are only available to people and services that require them, without exposing them more widely. Some configuration registry products and services can be used to store both secret and non-secret values. But it’s important to avoid storing secrets in registries that don’t protect them. Doing so makes the registry an easy target for attackers.

Related patterns

You probably need to pass at least one parameter to the stack tool to indicate which stack instance’s parameters to use. You can use either the stack provisioning script or pipeline stack parameter pattern for this.

Infrastructure automation tool registries

Many infrastructure automation toolchains include a configuration registry service. These tend to be part of a centralized service that may also include features such as source code management, monitoring, dashboards, and command orchestration. Examples of these include:

• Chef Infra Server

• PuppetDB

• Ansible Tower

• Salt Mine

You may be able to use these services with tools outside the toolchain that provides them. Most can expose values, so you could write a script that discovers information about the current state of infrastructure managed by the configuration tool. Some infrastructure tool registries are extensible, so you can use them to store the data from other tools.

However, this creates a dependency on whatever toolchain provides the registry service. The service may not fully support integration with third-party tools. It might not offer a contract or API that guarantees future compatibility.

So if you’re considering using an infrastructure tool’s data store as a general-purpose configuration registry, consider how well it supports this use case, and what kind of lock-in it creates.

General-purpose configuration registry products

There are many dedicated configuration registry and key-value store database products available outside the toolchains of a particular automation tool. Some examples include:

• Zookeeper

• etcd

• Consul2

• doozerd

These are generally compatible with different tools, languages, and systems, so avoid locking you into any particular toolchain.

However, it can take a fair bit of work to define how data should be stored. Should keys be structured like environment/service/application, service/application/environment, or something else entirely? You may need to write and maintain custom code to integrate different systems with your registry. And a configuration registry gives your team yet another thing to deploy and run.

Platform registry services

Most cloud platforms provide a key-value store service, such as the AWS SSM Parameter Store. These give you most of the advantages of a general-purpose configuration registry product, without forcing you to install and support it yourself. However, it does tie you to that cloud provider. In some cases, you may find yourself using a registry service on one cloud to manage infrastructure running on another!

Rather than running a configuration registry server, some teams build a custom lightweight configuration registry by storing configuration files in a central location, or by using distributed storage. They typically use an existing file storage service like an object store (e.g., an S3 bucket on AWS), a version control system, networked filesystem, or even a web server.