The Example Stack

To start, we can define a single reusable stack that has all of the infrastructure other than the web server cluster. The project structure could look like Example 9-1.

stack-project/
   └── src/
       ├── appserver_vm.infra
       ├── appserver_networking.infra
       ├── database.infra
       └── database_networking.infra

Within this project, the file appserver_vm.infra includes code along the lines of what is shown in Example 9-2.

virtual_machine:
  name: appserver-${customer}-${environment}
  ram: 4GB
  address_block: ADDRESS_BLOCK.appserver-${customer}-${environment}
  storage_volume: STORAGE_VOLUME.app-storage-${customer}-${environment}
  base_image: SERVER_IMAGE.shopspinner_java_server_image
  provision:
    tool: servermaker
    parameters:
      maker_server: maker.shopspinner.xyz
      role: appserver
      customer: ${customer}
      environment: ${environment}

storage_volume:
  id: app-storage-${customer}-${environment}
  size: 80GB
  format: xfs

A team member or automated process can create or update an instance of the stack by running the stack tool. They pass values to the instance using one of the patterns from Chapter 7.

As described in Chapter 8, the team uses multiple test stages (“Progressive Testing”), organized in a sequential pipeline (“Infrastructure Delivery Pipelines”).

Pipeline for the Example Stack

A simple pipeline for the ShopSpinner application infrastructure stack has two testing stages,¹ followed by a stage that applies the code to each customer’s production environment (see Figure 9-1).

Figure 9-1. Simplified example pipeline for a stack

The first stage of the pipeline is the stack build stage. A build stage for an application usually compiles code, runs unit tests (described in “Test Pyramid”), and builds a deployable artifact. See “Building an Infrastructure Project” for more details of a typical build stage for infrastructure code. Because earlier stages in a pipeline should run faster, the first stage is normally used to run offline tests.

The second stage of the example pipeline runs online tests for the stack project. Each of the pipeline stages may run more than one set of tests.

Syntax Checking

With most stack tools, you can run a dry run command that parses your code without applying it to infrastructure. The command exits with an error if there is a syntax error. The check tells you very quickly when you’ve made a typo in your code change, but misses many other errors. Examples of scriptable syntax checking tools include terraform validate and aws cloudformation validate-template.

$ stack validate

Error: Invalid resource type

  on appserver_vm.infra line 1, in resource "virtual_mahcine":

stack does not support resource type "virtual_mahcine".

Some tools can parse and analyze stack source code for a wider class of issues than just syntax, but still without connecting to an infrastructure platform. This analysis is often called linting.2 This kind of tool may look for coding errors, confusing or poor coding style, adherence to code style policy, or potential security issues. Some tools can even modify code to match a certain style, such as the terraform fmt command. There are not as many tools that can analyze infrastructure code as there are for application programming languages. Examples include tflint, CloudFormation Linter, cfn_nag, tfsec, and checkov.

Here’s an example of an error from a fictional analysis tool:

$ stacklint
1 issue(s) found:

Notice: Missing 'Name' tag (vms_must_have_standard_tags)

  on appserver_vm.infra line 1, in resource "virtual_machine":

In this example, we have a custom rule named vms_must_have_standard_tags that requires all virtual machines to have a set of tags, including one called Name.

Static Code Analysis with API

Depending on the tool, some static code analysis checks may connect to the cloud platform API to check for conflicts with what the platform supports. For example, tflint can check Terraform project code to make sure that any instance types (virtual machine sizes) or AMIs (server images) defined in the code actually exist. Unlike previewing changes (see “Preview: Seeing What Changes Will Be Made”), this type of validation tests the code in general, rather than against a specific stack instance on the platform.

The following example output fails because the code declaring the virtual server specifies a sever image that doesn’t exist on the platform:

$ stacklint
1 issue(s) found:

Notice: base_image 'SERVER_IMAGE.shopspinner_java_server_image' doesn't
  exist (validate_server_images)

  on appserver_vm.infra line 5, in resource "virtual_machine":

Testing with a Mock API

You may be able to apply your stack code to a local, mock instance of your infrastructure platform’s API. There are not many tools for mocking these APIs. The only one I’m aware of as of this writing is Localstack. Some tools can mock parts of a platform, such as Azurite, which emulates Azure blob and queue storage.

Applying declarative stack code to a local mock can reveal coding errors that syntax or code analysis checks might not find. In practice, testing declarative code with infrastructure platform API mocks isn’t very valuable, for the reasons discussed in “Challenge: Tests for Declarative Code Often Have Low Value”. However, these mocks may be useful for unit testing imperative code (see “Programmable, Imperative Infrastructure Languages”), especially libraries (see “Dynamically Create Stack Elements with Libraries”).

Preview: Seeing What Changes Will Be Made

Some stack tools can compare stack code against a stack instance to list changes it would make without actually changing anything. Terraform’s plan subcommand is a well-known example.

virtual_machine:
  name: myappserver
  base_image: "java_server_image"

Previewing stack changes is useful for checking a limited set of risks immediately before applying a code change to an instance. But it is less useful for testing code that you intend to reuse across multiple instances, such as across test environments for release delivery. Teams using copy-paste environments (see “Antipattern: Copy-Paste Environments”) often use a preview stage as a minimal test for each environment. But teams using reusable stacks (see “Pattern: Reusable Stack”) can use test instances for more meaningful validation of their code.

Verification: Making Assertions About Infrastructure Resources

Given a stack instance, you can have tests in an online stage that make assertions about the infrastructure in the stack. Some examples of frameworks for testing infrastructure resources include:

• Awspec

• Clarity

• Inspec

• Taskcat

• Terratest

A set of tests for the virtual machine from the example stack code earlier in this chapter could look like this:

given virtual_machine(name: "appserver-testcustomerA-staging") {
  it { exists }
  it { is_running }
  it { passes_healthcheck }
  it { has_attached storage_volume(name: "app-storage-testcustomerA-staging") }
}

Most stack testing tools provide libraries to help write assertions about the types of infrastructure resources I describe in Chapter 3. This example test uses a virtual_machine resource to identify the VM in the stack instance for the staging environment. It makes several assertions about the resource, including whether it has been created (exists), whether it’s running rather than having terminated (is_running), and whether the infrastructure platform considers it healthy (passes_healthcheck).

Simple assertions often have low value (see “Challenge: Tests for Declarative Code Often Have Low Value”), since they simply restate the infrastructure code they are testing. A few basic assertions (such as exists) help to sanity check that the code was applied successfully. These quickly identify basic problems with pipeline stage configuration or test setup scripts. Tests such as is_running and passes_healthcheck would tell you when the stack tool successfully creates the VM, but it crashes or has some other fundamental issue. Simple assertions like these save you time in troubleshooting.

Although you can create assertions that reflect each of the VM’s configuration items in the stack code, like the amount of RAM or the network address assigned to it, these have little value and add overhead.

The fourth assertion in the example, has_attached storage_volume(), is more interesting. The assertion checks that the storage volume defined in the same stack is attached to the VM. Doing this validates that the combination of multiple declarations works correctly (as discussed in “Testing combinations of declarative code”). Depending on your platform and tooling, the stack code might apply successfully but leave the server and volume correctly tied together. Or you might make an error in your stack code that breaks the attachment.

Another case where assertions can be useful is when the stack code is dynamic. When passing different parameters to a stack can create different results, you may want to make assertions about those results. As an example, this code creates the infrastructure for an application server that is either public facing or internally facing:

virtual_machine:
  name: appserver-${customer}-${environment}
  address_block:
    if(${network_access} == "public")
      ADDRESS_BLOCK.public-${customer}-${environment}
    else
      ADDRESS_BLOCK.internal-${customer}-${environment}
    end

You could have a testing stage that creates each type of instance and asserts that the networking configuration is correct in each case. You should move more complex variations into modules or libraries (see Chapter 16) and test those modules separately from the stack code. Doing this simplifies testing the stack code.

Asserting that infrastructure resources are created as expected is useful up to a point. But the most valuable testing is proving that they do what they should.

Outcomes: Proving Infrastructure Works Correctly

Functional testing is an essential part of testing application software. The analogy with infrastructure is proving that you can use the infrastructure as intended. Examples of outcomes you could test with infrastructure stack code include:

Testing outcomes is more complicated than verifying that things exist. Not only do your tests need to create or update the stack instance, as I discuss in “Life Cycle Patterns for Test Instances of Stacks”, but you may also need to provision test fixtures. A test fixture is an infrastructure resource that is used only to support a test (I talk about test fixtures in “Using Test Fixtures to Handle Dependencies”).

This test makes a connection to the server to check that the port is reachable, and returns the expected HTTP response:

given stack_instance(stack: "shopspinner_networking",
                     instance: "online_test") {

  can_connect(ip_address: stack_instance.appserver_ip_address,
              port:443)

  http_request(ip_address: stack_instance.appserver_ip_address,
              port:443,
              url: '/').response.code is('200')

}

The testing framework and libraries implement the details of validations like can_connect and http_request. You’ll need to read the documentation for your test tool to see how to write actual tests.

Test Doubles for Upstream Dependencies

When you need to test a stack that depends on another stack, you can create a test double. For stacks, this typically means creating some additional infrastructure. In our example of the shared network stack and the application stack, the application stack needs to create its server in a network address block that is defined by the network stack. Your test setup may be able to create an address block as a test fixture to test the application stack on its own.

A potential benefit of this type of decoupling is to make stacks more reusable and composable. The ShopSpinner team might want to create different network stack projects for different purposes. One stack creates tightly controlled and audited networking for services that have stricter compliance needs, such as payment processing subject to the PCI standard, or customer data protection regulations. Another stack creates networking that doesn’t need to be PCI compliant. By testing application stacks without using either of these stacks, the team makes it easier to use the stack code with either one.

Test Fixtures for Downstream Dependencies

You can also use test fixtures for the reverse situation, to test a stack that provides resources for other stacks to use. In Figure 9-3, the stack instance defines networking structures for ShopSpinner, including segments and routing for the web server container cluster and application servers. The network stack doesn’t provision the container cluster or application servers, so to test the networking, the setup provisions a test fixture in each of these segments.

Figure 9-3. Test instance of the ShopSpinner network stack, with test fixtures

The test fixtures in these examples are a pair of container instances, one assigned to each of the network segments in the stack. You can often use the same testing tools that you use for verification testing (see “Verification: Making Assertions About Infrastructure Resources”) for outcome testing. These example tests use a fictional stack testing DSL:

given stack_instance(stack: "shopspinner_networking",
                     instance: "online_test") {

  can_connect(from: $HERE,
              to: get_fixture("web_segment_instance").address,
              port:443)

  can_connect(from: get_fixture("web_segment_instance"),
              to: get_fixture("app_segment_instance").address,
              port: 8443)

}

The method can_connect executes from $HERE, which would be the agent where the test code is executing, or from a container instance. It attempts to make an HTTPS connection on the specified port to an IP address. The get_fixture() method fetches the details of a container instance created as a test fixture.

You can see the connections that the example test code makes in Figure 9-4.

Figure 9-4. Testing connectivity in the ShopSpinner network stack

The diagram shows the paths for both tests. The first test connects from outside the stack to the test fixture in the web segment. The second test connects from the fixture in the web segment to the fixture in the application segment.

Refactor Components So They Can Be Isolated

Sometimes a particular component can’t be easily isolated. Dependencies on other components may be hardcoded or simply too messy to pull apart. One of the benefits of writing tests while designing and building systems, rather than afterward, is that it forces you to improve your designs. A component that is difficult to test in isolation is a symptom of design issues. A well-designed system should have loosely coupled components.

There are several strategies for restructuring systems. Martin Fowler has written about refactoring and other techniques for improving system architecture. For example, the Strangler Application prioritizes keeping the system fully working while restructuring it over time.

Part IV of this book involves more detailed rules and examples for modularizing and integrating infrastructure.

Support Local Testing

People working on infrastructure stack code should be able to run the tests themselves before pushing code into the shared pipeline and environments. “Personal Infrastructure Instances” discusses approaches to help people work with personal stack instances on an infrastructure platform. Doing this allows you to code and run online tests before pushing changes.

As well as being able to work with personal test instances of stacks, people need to have the testing tools and other elements involved in running tests on their local working environment. Many teams use code-driven development environments, which automate installing and configuring tools. You can use containers or virtual machines for packaging development environments that can run on different types of desktop systems.4 Alternatively, your team could use hosted workstations (hopefully configured as code), although these may suffer from latency, especially for distributed teams.

A key to making it easy for people to run tests themselves is using the same test orchestration scripts across local work and pipeline stages. Doing this ensures that tests are set up and run consistently everywhere.

Avoid Tight Coupling with Pipeline Tools

Many CI and pipeline orchestration tools have features or plug-ins for test orchestration, even configuring and running the tests for you. While these features may seem convenient, they make it difficult to set up and run your tests consistently outside the pipeline. Mixing test and pipeline configuration can also make it painful to make changes.

Many teams write custom scripts to orchestrate tests. These scripts are similar to or may even be the same scripts used to orchestrate stack management (as described in “Using Scripts to Wrap Infrastructure Tools”). People use Bash scripts, batch files, Ruby, Python, Make, Rake, and others I’ve never heard of.

There are a few tools available that are specifically designed to orchestrate infrastructure tests. Two I know of are Test Kitchen and Molecule. Test Kitchen is an open source product from Chef that was originally aimed at testing Chef cookbooks. Molecule is an open source tool designed for testing Ansible playbooks. You can use either tool to test infrastructure stacks, for example, using Kitchen-Terraform.

The challenge with these tools is that they are designed with a particular workflow in mind, and can be difficult to configure to suit the workflow you need. Some people tweak and massage them, while others find it simpler to write their own scripts.