Motivation

Resource matching is straightforward to implement with most stack management tools and languages. The pattern mostly eliminates hardcoded dependencies, reducing coupling.

Resource matching also avoids coupling on tools. The provider infrastructure and consumer stack can be implemented using different tools.

Applicability

Use resource matching for discovering dependencies when the teams managing provider and consumer code both have a clear understanding of which resources should be used as dependencies. Consider switching to an alternative pattern if you experience issues with breaking dependencies between teams.

Resource matching is useful in larger organizations, or across organizations, where different teams may use different tooling to manage their infrastructure, but still need to integrate at the infrastructure level. Even where everyone currently uses a single tool, resource matching reduces lock-in to that tool, creating the option to use new tools for different parts of the system.

Consequences

As soon as a consumer stack implements resource matching to discover a resource from another stack, the matching pattern becomes a contract. If someone changes the naming pattern of the VLAN in the shared networking stack, the consumer’s dependency breaks.

So a consumer team should only discover dependencies by matching resources in ways that the provider team explicitly supports. Provider teams should clearly communicate what resource matching patterns they support, and be sure to maintain the integrity of those patterns as a contract.

Implementation

There are several ways to discover infrastructure resources by matching. The most straightforward method is to use variables in the name of the resource, as shown in the example code from earlier:

virtual_machine:
  name: "appserver-${ENVIRONMENT_NAME}"
  vlan: "vlan-appserver-${ENVIRONMENT_NAME}"

The string vlan-appserver-${ENVIRONMENT_NAME} will match the relevant VLAN for the environment.

Most stack languages have features to match other attributes than the resource name. Terraform has data sources and AWS CDK’s supports resource importing.

In this example (using pseudocode) the provider assigns tags to its VLANs:

vlans:
  - appserver_vlan
    address_range: 10.2.0.0/8
    tags:
      network_tier: "application_servers"
      environment: ${ENVIRONMENT_NAME}

The consumer code discovers the VLAN it needs using those tags:

external_resource:
  id: appserver_vlan
  match:
    tag: name == "network_tier" && value == "application_servers"
    tag: name == "environment" && value == ${ENVIRONMENT_NAME}

virtual_machine:
  name: "appserver-${ENVIRONMENT_NAME}"
  vlan: external_resource.appserver_vlan

Related patterns

The resource matching pattern is similar to the stack data lookup pattern. The main difference is that resource matching doesn’t depend on the implementation of the same stack tool across provider and consumer stacks.

Motivation

Stack management tool vendors make it easy to use their stack data lookup features to integrate different projects. Most implementations of data sharing across stacks require the provider stack to explicitly declare which resources to publish for use by other stacks. Doing this discourages consumer stacks from creating dependencies on resources without the provider’s knowledge.

Applicability

Using stack data lookup functionality to discover dependencies across stacks works when all of the infrastructure in a system is managed using the same tool.

Consequences

Stack data lookup tends to lock you into a single stack management tool. It’s possible to use the pattern with different tools, as described in the implementation of this pattern. But this adds complexity to your implementation.

This pattern sometimes breaks across different versions of the same stack tool. An upgrade to the tool may involve changing the stack data structures. This can cause problems when upgrading a provider stack to the newer version of the tool. Until you upgrade the consumer stack to the new version of the tool as well, it may not be possible for the older version of the tool to extract the resource values from the upgraded provider stack. This stops you from rolling out stack tool upgrades incrementally across stacks, potentially forcing a disruptive coordinated upgrade across your estate.

Implementation

The implementation of stack data lookup uses functionality from your stack management tool and its definition language.

Terraform stores output values in a remote state file. Pulumi also stores resource details in a state file that can be referenced in a consumer stack using a StackReference. CloudFormation can export and import stack output values across stacks, which AWS CDK can also access.²

The provider usually explicitly declares the resources it provides to consumers:

stack:
  name: shared_network_stack
  environment: ${ENVIRONMENT_NAME}

vlans:
  - appserver_vlan
    address_range: 10.2.0.0/8

export:
  - appserver_vlan_id: appserver_vlan.id

The consumer declares a reference to the provider stack and uses this to refer to the VLAN identifier exported by that stack:

external_stack:
  name: shared_network_stack
  environment: ${ENVIRONMENT_NAME}

virtual_machine:
  name: "appserver-${ENVIRONMENT_NAME}"
  vlan: external_stack.shared_network_stack.appserver_vlan.id

This example embeds the reference to the external stack in the stack code. Another option is to use dependency injection (“Dependency Injection”) so that your stack code is less coupled to the dependency discovery. Your orchestration script looks up the output value from the provider stack and passes it to your stack code as a parameter.

Although stack data lookups are tied to the tool that manages the provider stack, you can usually extract these values in a script, so you can use them with other tools, as shown in Example 17-1.

#!/usr/bin/env bash

VLAN_ID=$(
  stack value \
    --stack_instance shared_network-staging \
    --export_name appserver_vlan_id
)

This code runs the fictional stack command, specifying the stack instance (shared_network-staging) to look in, and the exported variable to read and print (appserver_vlan_id). The shell command stores the command’s output, which is the ID of the VLAN, in a shell variable named VLAN_ID. The script can then use this variable in different ways.

Related patterns

The main alternative patterns are resource matching (“Pattern: Resource Matching”) and registry lookup.

Issues with mixing dependency and definition code

Combining dependency discovery with stack definition code adds cognitive overhead to reading or working with the code. Although it doesn’t stop you from getting things done, the subtle friction adds up.

Coupling dependency management and definition can make it hard to create and test instances of the stack. Many approaches to testing use practices like ephemeral instances (“Pattern: Ephemeral Test Stack”) or test doubles (“Using Test Fixtures to Handle Dependencies”) to enable fast and frequent testing. This can be challenging if setting up dependencies involves too much work or time.

Hardcoding specific assumptions about dependency discovery into stack code can make it less reusable. For example, if you create a core application server infrastructure stack for other teams to use, different teams may want to use different methods to configure and manage their dependencies. Some teams may even want to swap out different provider stacks. For example, they might use different networking stacks for public-facing and internally facing applications.

Decoupling dependencies from their discovery

Dependency injection (DI) is a technique where a component receives its dependencies, rather than discovering them itself. An infrastructure stack project would declare the resources it depends on as parameters, the same as instance configuration parameters described in Chapter 7. A script or other tool that orchestrates the stack management tool (“Using Scripts to Wrap Infrastructure Tools”) would be responsible for discovering dependencies and passing them to the stack.

Consider how the application-infrastructure-stack example used to illustrate the dependency discovery patterns earlier in this chapter would look with DI:

parameters:
  - ENVIRONMENT_NAME
  - VLAN

virtual_machine:
  name: "appserver-${ENVIRONMENT_NAME}"
  vlan: ${VLAN}

The code declares two parameters that must be set when applying the code to an instance. The ENVIRONMENT_NAME parameter is a simple stack parameter, used for naming the application server virtual machine. The VLAN parameter is the identifier of the VLAN to assign the virtual machine to.

To manage an instance of this stack, you need to discover and provide a value for the VLAN parameter. Your orchestration script can do this, using any of the patterns described in this chapter. It might set the parameter based on tags, by finding the provider stack project’s outputs using the stack management tool, or it could look up the values in a registry.

An example script using stack data lookup (see “Pattern: Stack Data Lookup”) might use the stack tool to retrieve the VLAN ID from the provider stack instance, as in Example 17-1, and then pass that value to the stack command for the consumer stack instance:

#!/usr/bin/env bash

ENVIRONMENT_NAME=$1

VLAN_ID=$(
  stack value \
    --stack_instance shared_network-${ENVIRONMENT_NAME} \
    --export_name appserver_vlan_id
)

stack apply \
  --stack_instance application_infrastructure-${ENVIRONMENT_NAME} \
  --parameter application_server_vlan=${VLAN_ID}

The first command extracts the appserver_vlan_id value from the provider stack instance named shared_network-${ENVIRONMENT_NAME}, and then passes it as a parameter to the consumer stack application_infrastructure-${ENVIRONMENT_NAME}.

The benefit of this approach is that the stack definition code is simpler and can be used in different contexts. When you work on changes to the stack code on your laptop, you can pass whatever VLAN value you like. You can apply your code using a local API mock (see “Testing with a Mock API”), or to a personal instance on your infrastructure platform (see “Personal Infrastructure Instances”). The VLANs you provide in these situations may be very simple.

In more production-like environments, the VLAN may be part of a more comprehensive network stack. This ability to swap out different provider implementations makes it easier to implement progressive testing (see “Progressive Testing”), where earlier pipeline stages run quickly and test the consumer component in isolation, and later stages test a more comprehensively integrated system.