What’s on a Server

It’s useful to think about the various things on a server and where they all come from. One way of thinking of the stuff on a server is to divide it into software, configuration, and data. These categories, elaborated in Table 11-1, are useful for understanding the server’s configuration code and lifecycle.

The difference between configuration and data is whether automation tools automatically manage what’s inside the file. So even though a system log is essential for infrastructure, infrastructure code treats it as externally managed data. Similarly, if an application stores information like its accounts and preferences for its users in files on the server, the server configuration tool treats that file as data.

Where Things Come From

The software, configuration, and data that make up a server instance may be added when the server is created and configured, or when the server is changed. These elements have several possible sources, as shown in Figure 11-2.

Figure 11-2. Packages and other elements of a server instance

The elements shown in the diagram include the following:

Base operating system

The OS installation image could be a physical disk, an ISO file, or a stock server image provided by an IaaS platform. In some cases, an installer process is run to customize the installed server.

OS package repositories

The OS installation, or a post-installation configuration step, can run a tool that downloads and installs packages from one or more repositories. OS vendors usually provide a repository with packages they support. You can also use third-party repositories for open source or commercial packages. You can additionally host an internal repository for packages developed or curated in-house.

Language, framework, and other platform repositories

In addition to OS-specific packages, you may install packages for languages or frameworks like Ruby or Python. As with OS packages, you may pull packages from repositories managed by language vendors, third parties, or internal groups.

Nonstandard packages

Some vendors and in-house groups provide software with their own installers rather than using a standard package management tool, or else give instructions for installing their software manually.

Separate material

You may add or modify files outside of an application or component—for example, adding user accounts or setting local firewall rules.

Server Configuration Code

The first generation of Infrastructure as Code tools focused on automated server configuration. The following are some of the tools in this space:

• Ansible

• CFEngine

• Chef

• Puppet

• Salt Project

Server configuration code can be applied to a server instance as part of the provisioning process when it is created, as I’ll describe in “Creating and Provisioning a New Server Instance”. It can also be applied to prebuilt server images that are later used to create instances, as I’ll describe in “Configuring a New Server Instance”.

Server configuration tools organize code in modules, such as Ansible playbooks, Chef cookbooks, and Puppet modules. Teams can then define server roles to specify which code modules to apply to each server.

Server Roles

A team may run many types of servers for multiple purposes, including container cluster nodes, DNS servers, or database server nodes. Server infrastructure code tools support the concept of a server role to specify which code modules to apply to a given server. Server roles are the entry point for managing servers, whether creating a new server, specifying a server to provision as part of an infrastructure stack, or configuring automated server cluster management like an AWS Auto Scaling group.

In addition to specifying which server code modules to apply to a server, a role may set default parameter values for the modules to use. For example, you may create an application-server role:

role: application-server
  server_modules:
    - jboss
    - monitoring_agent
    - logging_agent
    - network_hardening
  parameters:
    - inbound_port: 8443

Assigning this role to a server applies the configuration for JBoss, monitoring and logging agents, and network hardening. It also sets a parameter for an inbound port, presumably so that the network_hardening module will open that port in the local firewall while locking down everything else. An email server may use the same module, but open port 25 for inbound SMTP connections.

Roles can become messy, so you should be deliberate and consistent in the way you use them. You may prefer to create fine-grained roles that you combine to compose specific servers. You assign multiple roles to a given server, such as ApplicationServer, MonitoredServer, and PublicFacingServer. Each of these roles includes a small number of server modules for a narrow purpose.

Alternatively, you may have higher-level roles, which include more modules. Each server probably has only one role, which may be quite specific (for example, ShoppingServiceApplicationServer or JenkinsServer).

A common approach is to use role inheritance. You define a base role that includes software and configuration common to all servers. This role might have networking hardening, administrative user accounts, and agents for monitoring and logging. More specific roles, such as for application servers or container hosts, include the base role and then add additional server configuration modules and parameters.

Creating a Server by Using Network Provisioning

In Chapter 3, I mentioned bare-metal provisioning tools for dynamically provisioning hardware servers. This process usually includes these steps:

1. Select an available physical server and trigger it to boot in a network install mode supported by the server firmware. This is typically a hardware-vendor-provided lights-out management (LOM) hardware function combined with the PXE boot standard for initiating the network booting process.

2. Boot the server from a minimal OS image to initiate the image installation.

3. Download a server image and copy it to the primary hard drive.

4. Reboot the server and apply server configuration or automated OS setup.

Tools to orchestrate the process include the following:

• Crowbar

• Cobbler

• Fully Automatic Installation (FAI)

• Foreman

• Canonical Metal as a Service (MAAS)

• RackN Digital Rebar

• Tinkerbell

In some cases, the server image may be an OS installation image. If so, the instance is booted from the image in step 4 in a mode that runs a scripted OS installer. Examples of scripted OS installers include Red Hat Kickstart, Solaris JumpStart, Debian Preseed, and the Windows installation answer file.

Creating an IaaS Server by Hand

IaaS platforms provide tools to create new servers via their APIs, which are usually web interfaces or command-line tools. Each time you create a new server, you select the options you want, including the source image, resources to allocate, and networking details:

$ mycloud server new \
    --source-image=stock-linux-1.23 \
    --memory=2GB \
    --vnet=appservers

The API may include a way to execute a provisioning script when booting it for the first time, which can be used to run server configuration code. Otherwise, the team will need to run the server configuration process after the server is created.

While it’s useful to play with UIs and command-line tools to experiment with a platform, it’s not a good way to create servers that people rely on. The same principles for creating and configuring stacks discussed previously apply to servers.² Setting options manually encourages mistakes and leads to inconsistently configured servers, unpredictable systems, and too much maintenance work.

Creating a Server from an Event

Most IaaS platforms will automatically create new server instances in specific circumstances. Two common cases are auto-scaling, adding servers to handle load increases, and auto-recovery, replacing a server instance when it fails. You normally define this by using stack code, as shown in Example 11-2.

server_cluster:
  server_instance:
    source_image: stock-linux-1.23
    memory: 2GB
    vnet: ${APPSERVER_VNET}
  scaling_rules:
    min_instances: 2
    max_instances: 5
    scaling_metric: cpu_utilization
    scaling_value_target: 40%
  health_check:
    type: http
    port: 8443
    path: /health
    expected_code: 200
    wait: 90s

This example tells the platform to keep at least two and at most five instances of the server running. The platform adds or removes instances as needed to keep the CPU utilization across them close to 40%.

The definition also includes a health check. This check makes an HTTP request to /health on port 8443, and considers the server to be healthy if the request returns a 200 HTTP response code. If the server doesn’t return the expected code after 90 seconds, the platform assumes the server has failed, and so destroys it and creates a new instance.

Baking Images and Frying Instances

Building configuration into server images (baking) optimizes for speed and ease of creating server instances. This process is especially useful for event-driven server creation like auto-scaling and recovery, when it’s important to get new servers into service quickly. Another advantage is that the image can be thoroughly tested offline with the confidence that servers created from it will behave consistently. Container cluster nodes are good candidates for using baked images. However, the lead time to get a change into production can be relatively slow when the change needs to be baked into an image and put through a testing pipeline.

Applying configuration when creating each server instance (frying) optimizes for variability among servers and for speed of delivering changes. When each server instance is configured or used differently, the information needed may be available only when the server is created. Servers created and customized on demand, such as to run customer-developed workloads, may be candidates for applying customized configuration parameters each time.

Often the most practical approach is to combine baking and frying. Create server images with as much as possible baked into them, especially larger items like language runtimes (the Java Development Kit, or JDK, comes to mind), application servers, database servers, and container cluster system software. Server infrastructure code can be applied with configuration supplied when creating servers to customize the configuration for particular uses and environments.

I’ve worked with teams that built images with their in-house software baked in but that used creation-time configuration scripts to fry in development builds of software for fast development and test feedback loops.

Pull Configuration with Initialization Scripts

Most IaaS platforms’ APIs for creating new server instances include a way to specify commands that will be executed the first time the server starts. AWS calls this user data, Azure calls it custom data, and Google Cloud uses startup scripts. You can configure this by using the API, command-line tool, infrastructure code, or the IaaS platform’s web interface. Options include directly passing a command to execute, files to upload, scripts to execute, parameters to pass, or a combination of these. The cloud-init standard tool is preinstalled on most Linux server images and offers advanced capabilities to configure new servers in combination with the data passed by the IaaS API.

Initialization scripts are often used to run server configuration tools like Ansible, Chef, or Puppet to configure the new server. The IaaS API passes a role name and a few parameters, such as the name of the environment the server is deployed into, and the tool downloads and applies the relevant infrastructure code. The server image usually has the server configuration tool agent preinstalled, although in some cases the initialization script might need to download and install the tool over the network before running it.

Teams that use more fully configured server images may not need to run a server configuration tool on new servers, but may instead run a script with a few simple commands to customize the specific server instance. This is especially common when using the immutable server approach.

Initialization scripts follow the pull configuration approach, so are also natural for teams that use continual synchronization to manage changes to servers and keep them consistent.

The advantage of configuring a new server by using an initialization script is that it isn’t necessary to open the server for external services or tools to connect over the network and run commands with root access. Server images can be more secure, possibly not even needing to run services like Secure Shell (SSH) at all.

Push Configuration with External Commands

An alternative to having the IaaS API execute commands on the server is to run a tool from an external agent that connects to the newly created server to run configuration commands over the network, usually using SSH. As I’ll discuss shortly, some server configuration tools, like Ansible, are designed to be run from a central service and connect into servers to configure them.

Teams may prefer using external tools and services to configure new servers because it avoids the need to preinstall server configuration agents, tools, or scripts. However, this approach requires ensuring that the configuration service is able to connect to the new server and execute commands as root, which potentially creates avenues for attackers to exploit.

Push on Change

The naive approach to using infrastructure code is to apply the code when you have a specific change you need to make. Many systems administration teams new to Infrastructure as Code follow the same pattern as when making changes manually, applying infrastructure code to only those systems they are aware need to be changed.

The drawback of this approach is that it leads to configuration drift among servers, with different versions of configuration and even system software across servers that should be broadly the same. For instance, newly created application servers may have the latest patches of the OS and application server software, while other application servers may be months or even years behind. In the worst case, this approach leads to snowflake servers, as described in Chapter 2. Pushing a change to only those servers that you know need it contributes to inconsistency and legacy systems.

Continuous Configuration Synchronization

Continuous configuration synchronization repeatedly and frequently applies configuration code to a server, whether or not the code has changed.

There are several reasons for doing this. One is to revert any changes that might have been made on a server outside of the configuration code. When code is applied only occasionally, some people might edit files on a server by hand to quickly fix an issue or because they aren’t aware of the automation. These conflicts are difficult to detect. The manual change will be reverted the next time the configuration is applied, but if it’s been a while, people may not notice or realize why something that had been working for a few weeks has broken.

Synchronizing configuration to servers on a short, automated cycle forces the discipline of including every change in the server code, breaking habits of making ad hoc tweaks. The rigor of this approach can help meet governance and compliance requirements such as those faced in financial services, government, and health care.

Malicious changes, such as those made by attackers gaining access to a server, may also be reverted. However, relying on continuous synchronization to stop attackers is not a good idea, since they might make changes to parts of the system not directly managed by code, or may even disable or subvert the automated updates on servers they’ve compromised.

Continuous synchronization could be run using push configuration, as described previously as an option for configuring new servers. A central service schedules commands that connect to each server and apply updates. As with configuring new servers, this approach requires enabling network access for the configuration service. It also requires ensuring that all servers are correctly registered with the service, to avoid any being left out of updates.

A more popular approach is pull configuration: an agent either runs as a service or is started by a scheduler like cron. The agent schedules a run, checking a central service or repository for the latest version of code and then applying it.

It’s a good idea to avoid running server configuration on the same schedule across a large estate of servers. The time to synchronize each should vary, perhaps with randomization, to avoid swamping the code repositories by running all the updates at the same time.³

Changing Servers by Replacement

Continuous synchronization leaves gaps in a system, in particular those parts of a server’s content that are not directly managed by infrastructure code. In theory, two servers built using the same starting image and with the same system and software package versions installed should be identical.

In practice, a server built fresh from the latest OS and packages often has differences from one built weeks or months earlier and brought to the same patch levels by updates. These differences might be subtle, but it’s not unusual for them to become the source of frustrating, inconsistent behavior.

And as I’ve mentioned earlier, people might make changes to areas of the system not directly managed by infrastructure code. Someone might decide to install a new package onto one server to try out some software, and leave it in place when they’re done or are interrupted by another task.

So the best way to ensure that a server is consistent and as deterministically managed as possible is to rebuild it. Many teams have processes to ensure that no server runs for very long before being rebuilt. “Very long” might be defined as a week or a few weeks.

The essential process for replacing a server instance is as follows:

1. Create a new instance without putting it into service.

2. Run checks to validate that the server is correct and ready.

3. Switch services to use the new server in place of the old one.

4. Allow monitoring to verify that the new server is handling its workload correctly.

5. Destroy the old server.

Depending on the applications and systems that use the server instance, it may be possible to carry out the replacement without downtime or at least with minimal downtime. You can find advice on this in Chapter 20, along with pointers for handling data that needs to be persistent.

Some IaaS platforms have functionality to automate the server replacement process. You can update the server image version used by a server cluster in a stack instance, and the platform automatically adds new instances and removes old ones on a rolling basis.

Immutable Server

The Immutable Server pattern is a strategy that uses replacement as the only way to make changes to a server.⁴ The entire server configuration is defined in the image used to create instances. A change is made by building a new version of the image and replacing all the server instances that use it.

Immutable servers reduce the risk of applying configuration changes to running servers. Every change is tested in an image delivery pipeline before being put into production.

Despite the name, content on immutable servers does change. Application data, logs, and running processes obviously do change while a server is running. The term immutable refers to the policy of not changing configuration or installed software or packages on a running server instance.

Hot-Cloning an Existing Server

Most infrastructure platforms make it simple to duplicate a running server, so some teams create new servers by selecting a server that is already in use and cloning it. This quick-and-easy approach gives you servers that are consistent at the point of cloning. However, it doesn’t create clean, consistently configured servers.

Cloned servers contain historical data from the original server, such as logs. This data pollution can make troubleshooting confusing. I’ve seen people spend time puzzling over error messages that turned out not to have come from the server they were debugging, but instead from the server it was cloned from.

Cloned servers aren’t consistent or reproducible, because even when you use the same server to make new ones, the source server is likely to have changed in subtle ways. The capability to build new servers that are each exactly the same, even when created later, is useful for troubleshooting and can also be important for compliance and auditing.

Cloning a production server can be useful for troubleshooting issues or experimenting, but clones should never be used for business-critical activities, including development and testing software.

Booting an OS Installer

Some IaaS platforms and most virtualization systems allow you to boot a new server directly from a standard bootable disk image such as an ISO file, which is effectively a snapshot of a CD or DVD. This feature makes it possible to boot a new server directly from an OS setup image, passing parameters or console commands to customize the OS installation and configuration process.

Installing a new OS and then customizing it gives team members more control over the images they create. They obviously still rely on the OS vendor or project team, but don’t need to worry about other third parties who may have interfered with prebuilt stock images.

Modifying a Stock Image

The most common approach for building server images is to boot a new server instance from an existing stock image for your IaaS platform, modify it, and save it as a new image. An existing image can save a lot of work over building a new image from a base OS installation, although it may have more packages and services than you need.

Orchestrating Image Building

You can use an orchestration tool to manage the process of building server images. The tool should let you specify the starting image, boot disk image, or other source, as well as the configuration tool and any scripts you want to run to configure your new image. You should define this process as code, for all the usual reasons.

Netflix pioneered the extensive use of in-house server images and released as open source the Aminator tool it created for that purpose.⁶

The most popular tool for building images is HashiCorp Packer. Packer supports every OS and IaaS or virtualization platform I’ve ever used or considered using. Some of the infrastructure platforms offer services for building their server images, including AWS EC2 Image Builder and Azure VM Image Builder.