Chapter 10. Designing Infrastructure Code Libraries

As described in Chapter 6, most stack infrastructure tools support code libraries. Terraform, OpenTofu, CloudFormation, and Bicep call them modules, CDK has constructs, and Pulumi has component resources. Organizing code into a library makes it possible to reuse the code across projects and to share it among teams.

As I’ve touched on before, Terraform (and OpenTofu modules are used widely because a module is the only component that the tool supports for versioning, distributing, and sharing code. This is why Terraform modules are often adapted into deployment components, as described in “Stack Module” and “Deployment Wrapper Stack”.

This chapter describes common patterns and antipatterns for designing and implementing code libraries.

Facade Module

The Facade Module pattern creates a simplified interface to a resource from the stack tool language or the infrastructure platform. The module exposes a few parameters to the calling code (Example 10-1).

Example 10-1. Code using a facade module

use module: foodspin-server
  name: checkout-appserver
  memory: 8GB

The module uses these parameters to call the resource it wraps, and it hardcodes values for other parameters needed by the resource (Example 10-2).

Example 10-2. Code for the example facade module

declare module: foodspin-server
  virtual_machine:
    name: ${name}
    source_image: hardened-linux-base
    memory: ${memory}
    provision:
      tool: servermaker
      maker_server: maker.foodspin.biz
      role: application_server
    network:
      vlan: application_zone_vlan

This example module allows the caller to create a virtual server, specifying its name and amount of memory. Every server created using the module uses a source image, role, and networking defined by the module.

Also Known As

Wrapper Module.

Motivation

A facade module simplifies and standardizes a common use case for an infrastructure resource. The stack code that uses a facade module should be simpler to write and easier to read. Improvements to the quality of the module code are rapidly available to all the stacks that use it.

Applicability

Facade modules work best for simple use cases, usually involving a basic infrastructure resource. They are also useful when the underlying API is complex and has options that aren’t necessary to expose.

Consequences

A facade module limits the way you can use the underlying infrastructure resource. This can be useful, simplifying options and standardizing around better and more secure implementations. But this approach limits flexibility, so it won’t apply to every use case.

A module is an extra layer of code between the stack code and the code that directly specifies the infrastructure resources. This extra layer adds at least some overhead to maintaining, debugging, and improving code. It can also make it harder for people to understand the stack code.

Implementation

Implementing the Facade Module pattern generally involves specifying an infrastructure resource with hardcoded values, and a small number of values that are passed through from the code that uses the module. A declarative infrastructure language is appropriate for a facade module.

Related Patterns

The Obfuscation Module pattern is a Facade Module that doesn’t hide much, adding complexity without adding much value. A Bundle Module declares multiple related infrastructure resources, so is like a Facade Module with more parts.

Obfuscation Module

The Obfuscation Module antipattern wraps the code for an infrastructure element defined by the stack language or infrastructure platform, but does not simplify it or add any particular value. In the worst cases, the module is harder to learn and understand than directly using the infrastructure code language. Example 10-3 shows code that uses a module called any_server.

Example 10-3. Code using an obfuscation module

use module: any_server
  server_name: checkout-appserver
  ram: 8GB
  source_image: base_linux_image
  provisioning_tool: servermaker
  server_role: application_server
  vlan: application_zone_vlan

The module itself passes the parameters directly to the stack management tool’s code, as shown in Example 10-4.

Example 10-4. Code for the example obfuscation module

declare module: any_server
  virtual_machine:
    name: ${server_name}
    source_image: ${origin_server_image}
    memory: ${ram}
    provision:
      tool: ${provisioning_tool}
      role: ${server_role}
    network:
      vlan: ${server_vlan}

Motivation

An obfuscation module may be a facade module gone wrong. Sometimes people write this kind of module aiming to follow the DRY principle. They see that code defining a common infrastructure element (such as a virtual server, load balancer, or security group) is used in multiple places in the codebase. So they create a module declaring that element type once and use the module everywhere instead of directly using the resource declaration code.

Other people create obfuscation modules in a quest to design their own language for referring to infrastructure elements, “improving” the one provided by their stack tool.

Applicability

Nobody intentionally writes an obfuscation module. You may debate whether a given module obfuscates or is a facade, and that debate is useful. You should consider whether a module adds real value and, if not, refactor it into code that uses the infrastructure tool’s language directly.

Consequences

Writing, using, and maintaining module code rather than directly using the constructs provided by your stack tool adds overhead. This approach adds more code to maintain, cognitive overhead to learn, and extra moving parts in your build and delivery process. A component should add enough value to make the overhead worthwhile.

Implementation

If a module neither simplifies the resources it defines nor adds value over the underlying stack language code, consider replacing usages by directly using the stack language.

If your goal is to simplify creating infrastructure from code for people who are not infrastructure developers, consider using higher-level abstractions, such as a stack or composition, instead of wrapping infrastructure resources in custom modules. The challenge for these users is rarely the syntax of the infrastructure coding language, but rather the low-level details of the IaaS resources that it defines.

Related Patterns

The Obfuscation Module antipattern is similar to the Facade Module but doesn’t noticeably simplify the underlying code.

Unshared Module

The Unshared Module antipattern is used only once rather than being reused by multiple stacks.

Motivation

People usually create unshared modules as a way to organize the code within a stack project.

Applicability

As a stack project’s code grows, you may be tempted to divide the code into modules. If you divide the code so that you can write tests for each module, working with the code can become easier. Otherwise, there may be better ways to improve the codebase.

Consequences

Organizing a single stack’s code into modules adds overhead and moving parts to the codebase, like versioning and dependency management. Building a reusable module when you don’t need to reuse it is an example of YAGNI, investing effort now for a benefit that you may or may not need in the future.

Implementation

When a stack project becomes too large, you have several alternatives to moving its code into modules. It’s often better to split the stack into multiple stacks, using an appropriate stack structural pattern from Chapter 7. If the stack is fairly cohesive, you could instead simply organize the code into multiple files and, if necessary, multiple folders. Doing this can make the code easier to navigate and understand without the overhead of the other options.

The rule of three for software reuse suggests that you should turn something into a reusable component when you find three places that you need to use it.¹

Bundle Module

The Bundle Module pattern declares a collection of related infrastructure resources with a simplified interface. The stack code uses the module to define what it needs to provision:

use module: application_server
  service_name: checkout_service
  application_name: checkout_application
  application_version: 1.23
  min_cluster: 1
  max_cluster: 3
  ram_required: 4GB

The module code declares multiple infrastructure resources, usually centered on a core resource. In Example 10-5, the resource is a server cluster but also includes a load balancer and DNS entry.

Example 10-5. Module code for an application server

declare module: application_server

  server_cluster:
    id: "${service_name}-cluster"
    min_size: ${min_cluster}
    max_size: ${max_cluster}
    each_server_node:
      source_image: base_linux
      memory: ${ram_required}
      provision:
        tool: servermaker
        role: appserver
        parameters:
          app_package: "${checkout_application}-${application_version}.war"
          app_repository: "repository.foodspin.biz"

  load_balancer:
    protocol: https
    target:
      type: server_cluster
      target_id: "${service_name}-cluster"

  dns_entry:
    id: "${service_name}-hostname"
    record_type: "A"
    hostname: "${service_name}.foodspin.biz"
    ip_address: {$load_balancer.ip_address}

Motivation

A bundle module is useful for defining a cohesive collection of infrastructure resources. It avoids verbose, redundant code. These modules capture knowledge about the various elements needed and how to wire them together for a common purpose.

Applicability

A bundle module may be suitable when you’re working with a declarative stack language and when the resources involved don’t vary across use cases. If you need the module to create different resources or to configure them differently depending on the usage, you should either create separate modules or switch to an imperative language and create an Infrastructure Domain Entity.

Consequences

A bundle module may provision more resources than you need in some situations. Users of the module should understand what it provisions and should avoid using the module if it’s overkill for their use case.

Implementation

Define the module declaratively, including infrastructure elements that are closely related to the declared purpose.

Related Patterns

The Facade Module pattern wraps a single infrastructure resource, while the Bundle Module includes multiple resources, although both are declarative in nature. The Infrastructure Domain Entity is similar to the Bundle Module but dynamically generates infrastructure resources. The Spaghetti Module antipattern is a bundle module that wishes it was a domain entity but descends into madness thanks to the limitations of its declarative language.

Spaghetti Module

The Spaghetti Module antipattern is configurable to the point where it creates significantly different results depending on the parameters given to it. The implementation of the module is messy and difficult to understand because it has too many moving parts; see Example 10-6.

Example 10-6. A spaghetti module

declare module: application-server-infrastructure
  variable: network_segment = {
    if ${parameter.network_access} = "public"
      id: public_subnet
    else if ${parameter.network_access} = "customer"
      id: customer_subnet
    else
      id: internal_subnet
    end
  }

  switch ${parameter.application_type}:
    "java":
      virtual_machine:
        origin_image: base_tomcat
        network_segment: ${variable.network_segment}
        server_configuration:
        if ${parameter.database} != "none"
          database_connection: ${database_instance.my_database.connection_string}
        end
        ...
    "NET":
      virtual_machine:
        origin_image: windows_server
        network_segment: ${variable.network_segment}
        server_configuration:
        if ${parameter.database} != "none"
          database_connection: ${database_instance.my_database.connection_string}
        end
        ...
    "php":
      container_group:
        cluster_id: ${parameter.container_cluster}
        container_image: nginx_php_image
        network_segment: ${variable.network_segment}
        server_configuration:
        if ${parameter.database} != "none"
          database_connection: ${database_instance.my_database.connection_string}
        end
        ...
  end

  switch ${parameter.database}:
    "mysql":
      database_instance: my_database
        type: mysql
        ...
    ...

This example code assigns the server it creates to one of three network segments, and optionally creates a database cluster and passes a connection string to the server configuration. In some cases, the code creates a group of container instances rather than a virtual server. This module is a bit of a beast.

Motivation

As with many other antipatterns, people create a spaghetti module by accident, often over time. They may create a facade module or a bundle module that grows in complexity to handle divergent use cases that seem similar on the surface.

Spaghetti modules often result from using a declarative language to implement the Infrastructure Domain Entity pattern.

Consequences

A module that does too many things is less maintainable than one with a tighter scope. The more things a module does, and the more variations that exist in the infrastructure that it can create, the harder it is to change it without breaking something. These modules are harder to test. Better-designed code is easier to test, so if you’re struggling to write automated tests and build pipelines to test the module in isolation, it’s a sign that you have a spaghetti module.

Implementation

A spaghetti module’s code often contains conditionals that apply different specifications in different situations. For example, a database cluster module might take a parameter to choose which database to provision.

When you realize you have a spaghetti module on your hands, you should refactor it. Often, you can split it into multiple modules, each with a more focused remit. For example, you might decompose your single application infrastructure module into multiple modules for different parts of the application’s infrastructure. An example of a stack that uses decomposed modules in this way, rather than using the spaghetti module shown previously, might look like Example 10-7.

Example 10-7. Using decomposed modules rather than a single spaghetti module

use module: java-application-servers
  name: checkout_appserver
  application: "shopping_app"
  application_version: "4.20"
  network_segment: customer_subnet
  server_configuration:
    database_connection: ${module.mysql-database.outputs.connection_string}

use module: mysql-database
  cluster_minimum: 1
  cluster_maximum: 3
  allow_connections_from: customer_subnet

Each of the modules is smaller, simpler, and easier to maintain and test than the original spaghetti module.

Related Patterns

The Spaghetti Module antipattern often results from an attempt at using declarative code to build an infrastructure domain entity. This antipattern could also result from a facade module or a bundle module that people tried to extend to handle different use cases.

Infrastructure Domain Entity

The Infrastructure Domain Entity pattern implements a high-level stack component by combining multiple lower-level infrastructure resources. An example of a higher-level concept is the infrastructure needed to run an application.

This example shows how a library that implements a Java application infrastructure instance might be used from stack project code:

use module: application_server
  service_name: checkout_service
  application_name: checkout_application
  application_version: 1.23
  traffic_level: medium

The code defines the application and version to deploy as well as a traffic level. The domain entity library code could look similar to the Bundle Module example but includes dynamic code to provision resources according to the traffic_level parameter:

...
  switch (${traffic_level}) {
    case ("high") {
      $appserver_cluster.min_size = 3
      $appserver_cluster.max_size = 9
    } case ("medium") {
      $appserver_cluster.min_size = 2
      $appserver_cluster.max_size = 5
    } case ("low") {
      $appserver_cluster.min_size = 1
      $appserver_cluster.max_size = 2
    }
  }
...

Motivation

A domain entity is often part of an abstraction layer that people can use to define and build infrastructure based on higher-level requirements. An infrastructure platform team builds components that other teams can use to assemble stacks.

Applicability

Because an infrastructure domain entity dynamically provisions infrastructure resources, it should be written in an imperative language rather than a declarative one.

Implementation

On a concrete level, implementing the Infrastructure Domain Entity pattern is a matter of writing the code. But the best way to create a high-quality codebase that is easy for people to learn and maintain is to take a design-led approach.

I recommend drawing from lessons learned in software architecture and design. The Infrastructure Domain Entity pattern derives from domain-driven design (DDD), which creates a conceptual model for the business domain of a software system and uses that to drive the design of the system itself.² Infrastructure, especially one designed and built as software, should be seen as a domain in its own right. The domain is building, delivering, and running software.

A particularly powerful approach is for an organization to use DDD to design the architecture for the business software, and then extend the domain to include the systems and services used for building and running that software.

Related Patterns

The Bundle Module pattern is similar to the Infrastructure Domain Entity in that it creates a cohesive collection of infrastructure resources. But a bundle module normally creates a fairly static set of resources, without much variation. The mindset of a bundle module is usually bottom-up, starting with the infrastructure resources to create. A domain entity is a top-down approach, starting with what’s required for the use case.

Most spaghetti modules for infrastructure stacks are a result of stretching declarative code to implement dynamic logic. But sometimes an infrastructure domain entity becomes overly complicated. A domain entity with poor cohesion becomes a spaghetti module.

Stack Module

The Stack Module pattern uses a library written to implement a complete, deployable stack. A separate wrapper stack (see “Deployment Wrapper Stack”) is implemented for each deployed instance of the module to provide the additional code and configuration necessary to deploy the module. Only one stack module is used for a deployed stack.

Also Known As

No-Code Module.

Motivation and Applicability

Teams use the Stack Module pattern when they want to redeploy the same code consistently across multiple instances, following the Reusable Stack pattern (“Reusable Stack”). Ideally, they would use a stack component for this purpose, but some tools (notably Terraform and OpenTofu) have strong support for modules as distributable, shareable, reusable components, but not for stacks.

Implementation

As mentioned in the description, many teams implement a wrapper stack project for each deployed instance of the stack module, setting configuration parameter values for the instance. Some tools or services, such as HashiCorp’s no-code provisioning, can automatically generate a wrapper stack project, so you can specify the stack module to deploy.

Alternatively, some teams prefer to write reusable stacks without modules, although this requires them to implement orchestration logic to distribute, share, and version their stack-level code.

Modular Monolith

The Modular Monolith antipattern is a monolithic stack that is divided into modules but that still suffers from the drawbacks of being a monolith.

Motivation

Some teams approach fixing a monolithic stack by breaking it into modules.

Consequences

Splitting a monolithic stack’s code into libraries may help make the code easier to understand and to change. However, this approach doesn’t solve the core problems of a monolith. Deploying the stack still takes too long, changes to one part risk impacting everything else in the stack, and the feedback loop for testing and delivering changes across environments is no faster.

Implementation

Rather than breaking a monolith into code libraries or modules, the team that owns it should find ways to split it into smaller, separately releasable and deployable stacks.

Related Patterns

Modular monoliths are a specific manifestation of a monolithic stack.

Conclusion

Code libraries can be useful for organizing, reusing, and sharing code. It’s important to understand the difference between a library or module and a deployable stack. In many cases it’s more useful to share code as a stack component rather than as a library, as described in “Reusable Stack”.

¹ The rule of three was defined in Robert Glass’s Facts and Fallacies of Software Engineering (Addison-Wesley). Jeff Atwood also commented on the rule of three in his post on the delusion of reuse.

² See Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans (Addison-Wesley).