Applying infrastructure code from the command line can be useful for a test instance of the infrastructure that nobody else uses. But running the tool from your local work environments creates problems with shared instances of infrastructure, whether it’s a production environment or a delivery environment (see “Delivery Environments”).

The person might make changes to their local version of the code before applying it. If they apply the code before pushing the changes to the shared repository, then nobody else has access to that version of the code. This can cause problems if someone else needs to debug the infrastructure.

If the person who applied their local version of the code does not immediately push their changes, someone else might pull and edit an older version of the code. When they apply that code, they’ll revert the first person’s changes. This situation quickly becomes confusing and hard to untangle (see Figure 20-2).

Figure 20-2. Locally editing and applying code leads to conflicts

Note that locking solutions, such as Terraform’s state locking, don’t prevent this situation. Locking stops two people from applying their code to the same instance simultaneously. But, as of this writing, locking solutions don’t stop people from applying divergent versions of code, as long as they each wait their turn.

So the lesson is that, for any instance of infrastructure, code should always be applied from the same location. You could designate an individual to be responsible for each instance. But this has many pitfalls, including a risky dependency on one person and their workstation. A better solution is to have a central system that handles shared infrastructure instances.

Applying Code from a Centralized Service

You can use a centralized service to apply infrastructure code to instances, whether it’s an application that you host yourself or a third-party service. The service pulls code from a source code repository or an artifact repository (see “Packaging Infrastructure Code as an Artifact”) and applies it to the infrastructure, imposing a clear, controlled process for managing which version of code is applied.

If two people pull and edit code, they must resolve any differences in their code when they integrate their code with the branch that the tool uses. When there is a problem, it’s easy to see which version of the code was applied and correct it (see Figure 20-3).

Figure 20-3. Centrally integrating and applying code

A central service also ensures that the infrastructure tool is run consistently, rather than assuming that a person doesn’t make a mistake or “improve” (deviate from) the workflow. It uses the same versions of the tool, scripts, and supporting utilities every time.

Using a central service aligns well with the pipeline model for delivering infrastructure code across environments (see “Infrastructure Delivery Pipelines”). Whatever tool or service you use to orchestrate changes through your pipeline takes on the responsibility for running your infrastructure tool to apply the latest version of code to each environment.

Another benefit of having a central service execute your infrastructure code is that it forces you and your team to automate the entire process. If you run the tool from your workstation, it’s easy to leave a few loose ends, steps that you need to do manually before or after running the tool. A central service gives you no choice other than making sure the task is entirely automated.

Source Code Branches in Workflows

Branches are a powerful feature of shared source repositories that make it easier for people to make changes to different copies of a codebase—branches—and then integrate their work when they’re ready. There are many strategies and patterns for using branches as part of a team’s workflow. Rather than elaborating on them here, I refer you to Martin Fowler’s article, “Patterns for Managing Source Code Branches”.

It’s worth highlighting a few distinctions of branching strategy in the context of Infrastructure as Code. One is the difference between path to production patterns and integration patterns for branching. The other is the importance of integration frequency.

Teams use path to production branching patterns to manage which versions of code to apply to environments.7 Typical path to production patterns include release branches and environment branches (environment branches are discussed in “Delivering code from a source code repository”).

Integration patterns for branching describes ways for people working on a codebase to manage when and how they integrate their work.8 Most teams use the mainline integration pattern, either with feature branching or continuous integration.

The specific pattern or strategy is less important than how you use it. The most important factor in the effectiveness of a team’s use of branches is integration frequency, how often everyone merges all of their code into the same (main) branch of the central repository.9 The DORA Accelerate research finds that more frequent integration of all of the code within a team correlates to higher commercial performance. Their results suggest that everyone in the team should integrate all of their code together—for example, to main or trunk—at least once a day.

Minimize Automation Lag

Automation lag is the time that passes between instances of running an automated process, such as applying infrastructure code. The longer it’s been since the last time the process ran, the more likely it will fail.10 Things change over time, even when nobody has consciously made a change.

Even if code hasn’t changed, applying it after a long gap in time can still fail, for various reasons:

• Someone has changed another part of the system, such as a dependency, in a way that only breaks when your code is reapplied.

• An upgrade or configuration change to a tool or service used to apply your code might be incompatible with your code.

• Applying unchanged code might nevertheless bring in updates to transitive dependencies, such as operating system packages.

• Someone may have made an manual fix or improvement that they neglected to fold back into the code. Reapplying your code reverts the fix.

The corollary to automation lag is, the more frequently you apply infrastructure code, the less likely it is to fail. When failures do occur, you can discover the cause more quickly, because less has changed since the last successful run.

Avoid Ad Hoc Apply

One habit that some teams carry over from Iron Age ways of working is only applying code to make a specific change. They might only use their infrastructure code to provision new infrastructure, but not for making changes to existing systems. Or they may write and apply infrastructure code to make ad hoc changes to specific parts of their systems. For example, they code a one-off configuration change for one of their application servers. Even when teams use code to make changes, and apply code to all instances, sometimes they may only apply code when they make a change to the code.

A core strategy for eliminating configuration drift is to continuously apply infrastructure code to instances, even when the code hasn’t changed. Many server configuration tools, including Chef and Puppet, are designed to reapply configuration on a schedule, usually hourly.11

The GitOps methodology (see “GitOps”) involves continuously applying code from a source code branch to each environment. You should be able to use a central service to apply code (as described in “Applying Code from a Centralized Service”) to continuously reapply code to each instance.

Immutable Infrastructure

Immutable infrastructure solves the problem of configuration drift in a different way. Rather than applying configuration code frequently to an infrastructure instance, you only apply it once, when you create the instance. When the code changes, you make a new instance and swap it out for the old one.

Making changes by creating a new instance requires sophisticated techniques to avoid downtime (“Zero Downtime Changes”), and may not be applicable to all use cases. Automation lag is still potentially an issue, so teams that use immutable infrastructure tend to rebuild instances frequently, as with phoenix servers.12

Reshuffling Responsibilities

Defining systems as code creates opportunities to reshuffle the responsibilities of the people involved in working on infrastructure (the people listed in “The People”) and the way those people engage with their work. Some factors that create these opportunities are:

Reuse

Infrastructure code can be designed, reviewed, and reused across multiple environments and systems. You don’t need a lengthy design, review, and signoff exercise for each new server or environment if you use code that has already been through that process.

Working code

Because code is quick to write, people can review and make decisions based on working code and example infrastructure. This makes for faster and more accurate feedback loops than haggling over diagrams and specifications.

Consistency

Your code creates environments far more consistently than humans following checklists. So testing and reviewing infrastructure in earlier environments gives faster and better feedback than doing these later in the process.

Automated testing

Automated testing, including for governance concerns like security and compliance, gives people working on infrastructure code fast feedback. They can correct many problems as they work, without needing to involve specialists for routine issues.

Democratize quality

People who aren’t specialists can make changes to the code for potentially sensitive areas of infrastructure, such as networking and security policies. They can use tools and tests created by specialists to sanity check their changes. And specialists can still review and approve changes before they’re applied to production systems. Reviewing this way is more efficient because the specialist can directly view code, test reports, and working test instances.

Governance channels

The infrastructure codebase, and pipelines used to deliver changes to production instances, can be organized based on their governance requirements. So a change to security policies goes through a review and signoff step not necessarily required for changes to less sensitive areas.

Chapter 8 explains principles and practices for implementing automated testing and pipelines to deliver code changes to environments. The term shift left describes how this impacts workflows and delivery practices.

Code is rigorously tested during implementation, at the “left” end of the flow shown in most process diagrams. So organizations can spend less time on heavyweight processes at the “right” end, just before applying code to production.

People involved in governance and testing focus on what happens during implementation, working with teams, providing tools, and enabling practices to test early and often.

An Example Process for Infrastructure as Code with Governance

ShopSpinner has a reusable stack (“Pattern: Reusable Stack”) that it can use to create the infrastructure for an application server to host an instance of its service for a customer. When someone changes the code for this stack, it can affect all of its customers.

Its technical leadership group, which is responsible for architectural decisions, defines CFRs (as described in “What Should We Test with Infrastructure?”) that the application server infrastructure must support. These CFRs include the number and frequency of orders that users can place on a customer instance, the response times for the interface, and recovery times for server failures.

The infrastructure team and the application team join with a pair of Site Reliability Engineers (SREs) and a QA to implement some automated tests that check the performance of the application server stack against the CFRs. They build these tests into the several stages of the pipeline, progressively testing different components of the stack (as per “Progressive Testing”).

Once the group has these tests in place, people don’t need to submit infrastructure changes for review by the technical leadership group, SREs, or anyone else. When an engineer changes the networking configuration, for example, the pipeline automatically checks whether the resulting infrastructure still meets the CFRs before they can apply it to production customer instances. If the engineer makes a mistake that breaks a CFR, they find out within minutes when a pipeline stage goes red and can immediately correct it.

In some cases, a change may cause a problem with a customer instance that isn’t caught by the automated tests. The group can conduct a blameless postmortem to review what happened. Perhaps the problem was that none of the CFRs covered the situation, so they need to change or add a CFR to their list. Or their testing may have a gap that missed the issue, in which case they improve the test suite.