Organizing Build Projects and Repositories

If you have multiple build projects, should you manage them all in a single repository or spread them across more than one? If you use more than one repository, should every project have its own repository, or should you group some projects into shared repositories? If so, how should you decide which projects to group together?

Consider the following design forces:

• Separating projects into their own repositories makes it easier to maintain boundaries at the code level.

• Having too many people working on code in a single repository can create friction and conflicts.

• Most source control systems manage permissions at the repository level, which may suggest organizing projects into repositories based on team ownership.

• Spreading code across multiple repositories can complicate working on changes that cross them.

• Code kept in the same repository is versioned and can be branched together, which simplifies some project integration and delivery strategies.

• Source code management systems (such as Git, Perforce, and Mercurial) have differing performance and scalability characteristics and features to support complex scenarios.

Let’s look at the main options for organizing projects across repositories in light of these factors.

Full codebase in one repository

Many teams manage an entire codebase in a single repository. A repository that contains a large codebase built together in a single process is a monorepo, as defined in the previous chapter. However, teams more commonly have multiple separate builds for different projects in their codebase managed in a single repository, as shown in Figure 16-1.

Figure 16-1. Building multiple projects in one repository

A pitfall of projects sharing a repository is that their boundaries can blur. It’s easy to have a component import code or other files from a different component’s folder structure, which then requires those components to be checked out and built together. Over time, code becomes tangled across projects and hard to maintain, because a change to a file in one project can have unexpected conflicts with others.

A separate repository for each build project

The other extreme is a microrepo, where each project is in a separate repository, as in Figure 16-2.

Figure 16-2. Each project in a separate repository

This strategy ensures a clean separation between projects, especially when you have a pipeline that builds and tests each project separately before integrating them. In theory, you could use build-time integration across projects managed in separate repositories, by first checking out all the builds (see Figure 16-3).

Figure 16-3. A single build across multiple repositories

However, building across multiple projects in a single repository is more practical, because their code is versioned together. Pushing changes for a single build to multiple repositories complicates the delivery process. The delivery stage would need some way to know which versions of all the involved repositories to check out to create a consistent build.

Single-project repositories work best when supporting delivery-time and apply-time integration. A change to any one repository triggers the delivery process for its project, bringing it together with other projects later in the flow.

Multiple repositories with multiple projects

While some organizations push toward one extreme or the other—single repository for everything, or a separate repository for each project—most maintain multiple repositories that each contain more than one build project (see Figure 16-4).

Figure 16-4. Multiple repositories with multiple projects

Often the grouping of projects into repositories happens organically rather than being driven by a strategy like monorepo or microrepo. However, a few factors influence how easy the projects are to work with.

One factor, as seen in the discussions of the other repository strategies, is the alignment of projects with their build and delivery strategy. Keep projects in a single repository when they are closely related, and especially when you integrate the projects at build time. Consider separating projects into separate repositories when their delivery paths aren’t tightly integrated.

Another factor is team ownership. Although multiple people and teams can work on different projects in the same repository, it can be distracting. Changelogs intermingle commit history from different teams with unrelated workstreams. Some organizations restrict access to code. Access control for source control systems is often managed by repository, which is another driver for deciding which projects go where.

As mentioned for single repositories, projects within a repository more easily become tangled with file dependencies. So teams might divide projects into repositories based on where they need stronger boundaries from an architectural and design perspective.

Organizing Types of Code

Different projects in an infrastructure codebase usually define different types of elements of your system, such as applications, infrastructure stacks, server configuration modules, and libraries. Any of these projects could include multiple types of code, including infrastructure code, configuration values, tests, and utility scripts. Having a strategy for organizing these types helps keep your codebase maintainable.

Several design forces should be considered when deciding how to organize code in a repository. I mentioned a few of these design forces in Chapter 5. One is to organize projects around team ownership, which helps reduce friction that comes when you fall afoul of Conway’s law. Another is considering the working sets, making it easy for people to work on the code for the groups of projects they tend to change together. A third is delivery scope, aligning project code and pipeline design.

Design forces also affect how to organize source code within a project. Using domains to drive design is usually good, but for infrastructure code there are different domains you might consider.

   └── src/
       ├── cluster.infra
       ├── database.infra
       ├── load_balancer.infra
       ├── routing.infra
       ├── firewall_rules.infra
       └── policies.infra

   └── src/
       ├── customer_service.infra
       ├── search_service.infra
       ├── menu_service.infra
       ├── shared_compute.infra
       └── shared_network.infra

   ├── build.sh
   ├── deploy.sh
   ├── src/
   ├── tests/
   ├── environments/
   └── pipeline/

Local IaaS Emulators

Developers can use various tools to emulate their IaaS platform, or parts of it, so they can apply code. Chapter 18 discusses some of these tools in detail. For local work, emulators may be handy for running some automated tests. However, most emulators don’t fully replicate the IaaS platform, because they don’t replicate fully usable infrastructure resources. Instead, they emulate the API endpoints of the IaaS platform. The infrastructure stack tool interacts with these endpoints when it deploys the code to the emulator, but the emulator doesn’t provision real infrastructure resources.

The emulator might maintain some stateful data structures that later calls to the API can query. Most emulators don’t have a particularly useful UI for developers to interact with, so they’re more useful for automated tests and validations rather than for interactive exploration of what your code does. That said, IaaS emulators can give you fast feedback, so running tests against them as you work is useful to catch basic errors.

Personal IaaS Environments

Given the limitations of local IaaS emulators, infrastructure developers will almost certainly find it useful to be able to apply their local code to infrastructure hosted on the real IaaS platform.

Some teams have multiple developers share an environment to test their infrastructure changes. However, sharing environments invites clashes and confusion. I’ve seen people spend hours debugging what they assumed was an error in their code or a flaky system, which turned out to be another developer testing their own changes to the same infrastructure. This leads to situations like a developer wondering, “Why do my changes to the load balancer keep disappearing?”

Each person working on infrastructure code should be able to provision their own instance of the infrastructure on the IaaS platform. Some useful metrics for the effectiveness of your infrastructure development process include measuring how quickly someone can provision a personal environment to work on, and utilization levels for these environments. High-performing infrastructure teams provision environments on demand and tear them down when they’re not in use.

Developers might provision personal environments by using a script or tool. Using the same tools and process that are used to deploy infrastructure in other environments in the pipeline through to production helps keep results consistent. Some teams manage personal environments by using the same hosted tools or services used in the pipeline. Each developer creates a branch of the code they’re working on, and the service deploys code from the branch. The developer commits their changes to the pipeline by merging their branch to the main codebase branch.

An advantage of deploying personal environments from the repository rather than from code checked out to a local workstation is that the code is available to others. I’ve seen a team struggle to tear down a personal instance that someone left running when they went on vacation, because they didn’t have access to the person’s local code.

Just Enough Environment

A common reason for having developers share an environment to test their local code changes is that it’s too expensive to provide every developer with their own environment. Ensuring that environments are provisioned on demand and destroyed when not in use helps to some extent. But sometimes an environment is so large that creating a new one takes too long.

Teams should aim to be able to provision a partial environment with just enough infrastructure to work on the change at hand. Throughout this book I’ve advocated for breaking the infrastructure for an environment into separately deployable stacks and infrastructure compositions. One of the many benefits of a composable infrastructure architecture is that it simplifies provisioning partial environments for development and testing.

A challenge with spinning up only some parts of an environment is that those parts may depend on infrastructure provided by parts of the environment that you don’t particularly need. You might need to test an infrastructure stack that can be spun up in under 2 minutes but find that its chain of dependencies takes nearly 30 minutes to provision.

You can create and use test fixtures to loosen the coupling between parts of your environment (see “Use Test Fixtures to Handle Dependencies”). For example, your networking stack may be fully featured with comprehensive hardening and audit logging. But the infrastructure component you’re testing may need only a subnet from that stack. You can create a lightweight networking stack that includes the bare minimum to test dependent components.

Pipeline Stage Content

A pipeline stage takes input material. A build stage’s input is usually source code for the component. Later stages will take the build as a package or repository branch or tag. Materials can also include code libraries to assemble into a build package, tests to execute, or configuration values.

Clearly defining the scope of the stage is useful, in terms of which parts of the system are being tested or reviewed. The stage may have dependencies on other parts of the system, such as an IaaS platform or APIs of a hosted service. The pipeline stage for a component should not be responsible for ensuring that its dependencies have been built correctly, instead proving whether they work as needed in combination with the component being tested.

When the pipeline stage completes, whether it succeeds or fails, it usually produces output material. The output for a build stage is the code or package to be distributed in later stages. These may be altered in some ways, such as added files, a version number, or a tag indicating the results of the stage. Most stages produce additional output materials like test reports or logs.

Pipeline Stage Context

In Chapter 17, I describe progressive testing strategies, where tests with smaller scope are run in earlier stages for faster feedback, with the scope of components under test increasing further along the pipeline.

In addition to progressively increasing the scope of the components tested in each stage of the pipeline, each stage may expand the environment and integrations the component is tested with. Earlier stages of the pipeline will run faster in a more minimal environment, and will run more reliably by integrating with test fixtures rather than live API services. Figure 16-7 illustrates progressive context for multiple pipeline stages.

The first stage in the pipeline is an offline stage; the infrastructure component is validated without deploying the code to the IaaS platform. The code executes entirely in a pipeline service agent, perhaps using an IaaS emulator. Any dependencies the infrastructure requires from other infrastructure are replaced with test fixtures such as a mock service.

Figure 16-7. Progressive pipeline stages

The second stage deploys the code on the IaaS platform but still uses test fixtures in place of dependencies. The test fixtures should be faster to deploy than the real infrastructure, which makes the tests run faster. The use of fixtures also keeps the tests in this stage focused on the component under test. The tests won’t fail because of issues in its dependencies, but only because of problems with the component itself.

The third stage closely replicates the production environment, deploying the infrastructure code on the IaaS platform and integrating with infrastructure and services it depends on. The testing in this stage should cover only issues that will emerge in this situation, rather than checking for issues that could be caught in the earlier stages.

“Use Test Fixtures to Handle Dependencies” discusses mocks and other text fixtures in more detail.

Delivery Pipeline Software and Services

Infrastructure pipelines need capabilities to implement, including a source code repository, (potentially) an artifact repository, pipeline orchestration, and a deployment service.

A pipeline orchestration system manages the flow of code and artifacts through the stages of a pipeline. Some systems model the end-to-end flow of a pipeline—​for example, Buildkite, Concourse, GoCD, and Spinnaker. These services are designed to build and deliver software but can deliver any type of code including infrastructure.

You can also build pipelines by using a build server such as Jenkins, TeamCity, Bamboo Data Center, or GitHub Actions. These tools are often job oriented rather than flow oriented. The core design doesn’t inherently correlate versions of code, artifacts, and runs across multiple jobs. Most of these products have added support for pipelines as an overlay in their UI and configuration.

Some services and tools are designed specifically for deploying Infrastructure as Code. Most of these are focused on deploying to a single instance of infrastructure as opposed to delivering code across multiple environments, so I cover them in Chapter 19.