Most infrastructure platforms provide a managed cluster service. You can define, provision, and change a cluster as code using your stack configuration tool. Using a cluster as a service is a matter of creating a stack that includes the cluster and supporting elements.

Many of these clusters are based on Kubernetes, including EKS, AKS, and GKE. Others are based on a proprietary container service, such as ECS.

Many teams prefer to install and manage application clusters themselves, rather than using managed clusters. Kubernetes is the most popular core cluster solution.

You can use an installer such as kops, Kubeadm, and kubespray to deploy Kubernetes onto infrastructure you’ve provisioned for it.

There are also packaged Kubernetes distributions that bundle other services and features. There are literally dozens of these, some of which are:1

• HPE Container Platform (Hewlett Packard)

• OpenShift (Red Hat)

• Pivotal Container Services (PKS) (VMware/Pivotal)

• Rancher RKE

Some of these products started out with their own container formats and application scheduling and orchestration services. Many decided to rebuild their core products around the overwhelmingly popular Docker and Kubernetes rather than to continue developing their own solutions.

But a handful of products have resisted assimilation into the Kubernetes borg. These include HashiCorp Nomad and Apache Mesos, both of which can orchestrate container instances as well as noncontainerized applications across different compute resources. The Cloud Foundry Application Runtime (CFAR) has its own container orchestration (Diego), although it can also be used with Kubernetes.2

Monolithic Stack Using Cluster as a Service

The simplest design is to define all of the parts of your cluster in a single stack, following the monolithic stack antipattern (see “Antipattern: Monolithic Stack”). Although monoliths become an antipattern at scale, a single stack can be useful when starting out with a small and simple cluster.

Example 14-1 uses a cluster as a service, similar to AWS EKS, AWS ECS, Azure AKS, and Google GKE. So the code defines the cluster, but it doesn’t need to provision servers to run the cluster management, because the infrastructure platform handles that behind the scenes.

address_block:
  name: cluster_network
  address_range: 10.1.0.0/16"
  vlans:
    - vlan_a:
        address_range: 10.1.0.0/8
    - vlan_b:
        address_range: 10.1.1.0/8
    - vlan_c:
        address_range: 10.1.2.0/8

application_cluster:
  name: product_application_cluster
  address_block: $address_block.cluster_network

server_cluster:
  name: "cluster_nodes"
  min_size: 1
  max_size: 3
  vlans: $address_block.cluster_network.vlans
  each_server_node:
    source_image: cluster_node_image
    memory: 8GB

This example leaves out many of the things you’d have for a real cluster, like network routes, security policies, and monitoring. But it shows that the significant elements of networking, cluster definition, and server pool for host nodes are all in the same project.

Monolithic Stack for a Packaged Cluster Solution

The code in Example 14-1 uses an application cluster service provided by the infrastructure platform. Many teams instead use a packaged application cluster solution (as described in “Packaged Cluster Distribution”). These solutions have installers that deploy the cluster management software onto servers.

When using one of these solutions, your infrastructure stack provisions the infrastructure that the installer needs to deploy and configure the cluster. Running the installer should be a separate step. This way, you can test the infrastructure stack separately from the application cluster. Hopefully, you can define the configuration for your cluster as code. If so, you can manage that code appropriately, with tests and a pipeline to help you to deliver updates and changes easily and safely.

It may be useful to use server configuration code (as in Chapter 11) to deploy your cluster management system onto your servers. Some of the packaged products use standard configuration tools, such as Ansible for OpenShift, so you may be able to incorporate these into your stack building process. Example 14-2 shows a snippet that you would add to the monolithic stack code in Example 14-1 to create a server for the cluster management application.

virtual_machine:
  name: cluster_manager
  source_image: linux-base
  memory: 4GB
  provision:
    tool: servermaker
    parameters:
      maker_server: maker.shopspinner.xyz
      role: cluster_manager

The code configures the server by running the fictitious servermaker command, applying the cluster_manager role.

Since there is only one stack, a single pipeline can test and deliver code changes to instances of the application cluster. However, there are other elements involved, including the server image for the host nodes and the applications themselves. Figure 14-3 shows a potential design for these pipelines.

Figure 14-3. An example of pipelines for a cluster using a monolithic stack

The top pipeline (Figure 14-4) builds a server image for the host nodes, as described in “Using a Pipeline to Test and Deliver a Server Image”. The result of this pipeline is a server image that has been tested in isolation. The tests for that image probably check that the container management software is installed and that it complies with security policies.

Figure 14-4. The pipeline for the host node server image

The bottom pipeline (Figure 14-5) is for an application that deploys onto the cluster. In practice, you’ll have a number of these, one for every separately deployed application. This pipeline includes at least one early stage to build and test the application on its own. It then has stages that deploy the application to the cluster in each environment. The application can be tested, reviewed, and made available for production use in these environments. The pipelines for the applications are very loosely coupled with the pipelines for the cluster instances. You may choose to trigger application testing stages after cluster updates. Doing this helps you to find any issues that the change to the cluster causes for the application by running the application-specific tests.

Figure 14-5. Pipelines for delivering applications to the cluster

The pipeline for the application cluster stack in Figure 14-6 starts with an offline stage (“Offline Testing Stages for Stacks”) that runs some syntax checking, and applies the stack code to a local mock of the infrastructure platform (“Testing with a Mock API”). These tests can catch problems at the coding level, without needing to use infrastructure platform resources, so they run quickly.

Figure 14-6. The pipeline for the cluster stack code

The second stage of this pipeline is an online stage (see “Online Testing Stages for Stacks”), and creates an instance of the stack on the infrastructure platform. The instance might be persistent (see “Pattern: Persistent Test Stack”) or ephemeral (see “Pattern: Ephemeral Test Stack”). The tests in this stage can check that the cluster management services are correctly created and accessible. You can also test for security issues—for instance, making sure that access to cluster management endpoints is locked down.3

Because this monolithic cluster stack includes the code to create the host node servers, the online testing stage can test these as well. A test could deploy a sample application to the cluster and prove that it’s working. The advantage of using a sample application rather than a real application is that you can keep it simple. Strip it down to a minimal set of dependencies and configurations, so you can be sure any test failures are caused by issues with the cluster provisioning, rather than due to complexities of deploying a real-world application.

Note that this pipeline stage is heavy. It tests both the cluster configuration and the host node server cluster. It also tests the server image in the context of the cluster. There are plenty of different things that can cause this stage to fail, which complicates troubleshooting and fixing failures.

Most of the elapsed time for this stage will almost certainly be taken up by provisioning everything, far more than the time to execute the tests. These two issues—the variety of things tested in this one stage, and the time it takes to provision—are the main drivers for breaking the cluster into multiple stacks.

Start by pulling the host node server pool into a separate stack, as in Example 14-3. You can provision and test an instance of this stack without the application cluster. Test that the platform successfully boots servers from your images and that network routes work correctly. You can also test reliability, triggering a problem with one of the servers and proving whether the platform automatically replaces it.

server_cluster:
  name: "cluster_nodes"
  min_size: 1
  max_size: 3
  vlans: $address_block.host_node_network.vlans
  each_server_node:
    source_image: cluster_node_image
    memory: 8GB

address_block:
  name: host_node_network
  address_range: 10.2.0.0/16"
  vlans:
    - vlan_a:
        address_range: 10.2.0.0/8
    - vlan_b:
        address_range: 10.2.1.0/8
    - vlan_c:
        address_range: 10.2.2.0/8

This code adds separate VLANs for the host nodes, unlike the earlier code for the monolithic stack (see Example 14-1). It’s good practice to split the host nodes and the cluster management into different network segments, which you could do within the monolithic stack. Breaking the stacks apart leads us to do this, if only to reduce coupling between the two stacks.

Breaking the stacks apart adds a new pipeline for the host node cluster stack, as shown in Figure 14-7.

Figure 14-7. Added pipeline for the host node pool

Although there are a few more stages in this combined pipeline, they are lighter and faster. The online test stage for the cluster management stack (highlighted in Figure 14-8) only provisions the cluster management infrastructure, which is faster than the online stage in the monolithic stack pipeline. This stack no longer depends on the pipeline for host node server images, and doesn’t include the server nodes. So the tests in this stage can focus on checking that cluster management is configured and secured correctly.

Figure 14-8. Online test stages for the cluster pipelines

This revised design joins the pipeline for the host node server stack together with the pipeline for the cluster management stack in a stack integration stage, as shown in Figure 14-9.

Figure 14-9. Stack integration test stage for a cluster

This is an online test stage that provisions and tests instances of both stacks together. These tests can focus on issues that only emerge with this combination, so shouldn’t duplicate testing activities from previous stages. This is the stage where you would deploy a sample application and prove that it runs correctly on the cluster. You could also test reliability and scaling by triggering failures in your test application and creating the conditions needed to add additional instances.

You might decide to split this into more stacks; for example, breaking the common networking infrastructure out of the management stack. Chapters 15 and 17 go into more details of decomposing and integrating infrastructure across stacks.

You can run different clusters for different parts of your software delivery process. Doing this could be as simple as running one cluster per environment (see Figure 14-10).

Figure 14-10. Pipelines managing one cluster for each application deployment environment

A dedicated cluster for every environment avoids inconsistencies you might get from having applications from multiple environments sharing resources. However, it may be difficult and expensive to maintain a separate instance for every delivery stage. For instance, if your delivery process dynamically creates test instances, you may need to dynamically create cluster instances to run them, which can be very slow.

A variation of separating clusters by delivery stage is to share clusters across multiple stages. For example, you might use different clusters based on governance requirements. In Figure 14-11, there are three clusters. The DEV cluster is for development, running instances where people create and use various data sets for more exploratory testing scenarios. The NON-PROD cluster is for more rigorous delivery stages, with managed test data sets. The PROD cluster hosts the PREPROD and PROD environments, both of which contain customer data and so have more rigorous governance requirements.

Figure 14-11. Cluster instances shared across multiple environments

When hosting multiple environments in a shared cluster, you should aim to keep each environment segregated as much as possible. Ideally, applications and operational services should not be able to see or interact with instances from other environments. Many of the mechanisms that application clustering solutions provide for separating applications are “soft.” For instance, you may be able to tag instances to indicate the environment, but this is purely a convention. You should look for stronger methods to segregate applications.

Clusters for Governance

One of the benefits of having separate clusters for different parts of the delivery process is that the governance requirements are usually different for different stages in the process. Production has tighter requirements because services running there are the most business-critical, and the data is the most sensitive.

Quite often, different parts of a system have different governance and compliance requirements that cut across delivery stages. The most common example is services that handle credit card numbers, which are subject to PCI standards. Other examples include services that deal with customer personal data, which may be subject to regimes such as GDPR.

Hosting services that are subject to stricter standards on dedicated clusters can simplify and strengthen compliance and auditing. You can impose stronger controls on these clusters, applications running on them, and delivery of code changes to them. Clusters hosting services with less strict compliance requirements can have streamlined governance processes and controls.

As an example, you could have two clusters, one used for development, testing, and production hosting for regulated services, and one for the unregulated services. Or you may split cluster instances by delivery stage and by regulation requirements, as illustrated in Figure 14-12.

Figure 14-12. Separate clusters for delivery stage and regulation requirements

Clusters for Teams

Yet another factor for organizing multiple clusters is team ownership. Often, different teams are responsible for delivering and running different types of applications and services, which may have different hosting requirements. For example, a team owning customer-facing services may have different requirements for governance and availability than a team that owns services for an internal department. A cluster allocated to a team can be optimized for the requirements of that team and its applications.

A service mesh is a decentralized network of services that dynamically manages connectivity between parts of a distributed system. It moves networking capabilities from the infrastructure layer to the application runtime layer of the model described in “The Parts of an Infrastructure System”. In a typical service mesh implementation, each application instance delegates communication with other instances to a sidecar process (see Figure 14-13).

Figure 14-13. Sidecars enable communication with other processes in a service mesh

Some of the services that a service mesh can provide to applications include:

Routing

Direct traffic to the most appropriate instance of a given application, wherever it is currently running. Dynamic routing with a service mesh enables advanced deployment scenarios, such as blue-green and canary, as described in “Changing Live Infrastructure”.

Availability

Enforce rules for limiting numbers of requests; for example, circuit breakers.

Security

Handle encryption, including certificates.

Authentication

Enforce rules on which services can connect to which. Manage certificates for peer-to-peer authentication.

Observability, monitoring, and troubleshooting

Record connections and other events so that people can trace requests through complex distributed systems.

Some examples of service meshes include HashiCorp Consul, Envoy, Istio, and Linkerd.

Service meshes are most commonly associated with containerized systems. However, you can implement the model in noncontainerized systems; for example, by deploying sidecar processes onto virtual machines.

A service mesh adds complexity. As with cloud native architectural models like microservices, a service mesh is appealing because it simplifies the development of individual applications. However, the complexity does not disappear; you’ve only moved it out into the infrastructure. So your organization needs to be prepared to manage this, including being ready for a steep learning process.

It’s essential to keep clear boundaries between networking implemented at the infrastructure level, and networking implemented in the service mesh. Without a good design and implementation discipline, you may duplicate and intermingle concerns. Your system is harder to understand, riskier to change, and harder to troubleshoot.