$ node -e "throw new Error()" ; echo $?

In this case, you should see a stack trace printed, followed by the number 1 on a line of its own. Thrown errors warrant a bit more discussion.

Exceptions, Rejections, and Emitted Errors

Before discussing errors too much, it’s useful to define a few terms—especially since they’re often conflated:

When an exception is thrown or a promise is rejected, it needs to be handled in some manner. When completely ignored it leads to an application crash—for example, an uncaught error will crash a Node.js process. Swallowing errors is universally a bad practice and will come back to bite you. However, checking if a specific anticipated error is thrown before swallowing it isn’t necessarily the end of the world.

Consider the following example of a swallowed error:

const lib = require('some-library');
try {
  lib.start();
} catch(e) {} // Sometimes lib throws even though it works
lib.send('message');

For example, you might find that when the some-library gets upgraded, it begins throwing another error, one that is a big deal. This usually leads to hours of debugging before finally finding the source of the underlying issue. For this reason, swallowing all errors is bad.

To swallow only a specific error, you might instead change the code to look like this:

catch(e) {
  if (e instanceof lib.Errors.ConnectionFallback) {
    // swallow error
  } else {
    throw e; // re-throw
  }
}

In this case, the exception is only swallowed if it is a specific error instance, otherwise it is rethrown again. This particular example assumes that a library author was thoughtful enough to export subclassed error instances. Unfortunately this often isn’t the case (not to mention instanceof checks can be tricky with a complex npm package hierarchy). Sometimes a library author might subclass errors but not export them. In those cases, you can check the .name field, for example by using e.name === 'ConnectionFallback'.

The most common approach for targeting specific errors is to parse the actual .message field. When doing this, your application will need to inspect text meant for human consumption—for example, using e.message.startsWith('Had to fallback'). This is unfortunately quite error prone! Error messages often have typos, and well-meaning contributors make PRs to fix them all the time. Such updates are usually released as a Semver patch release and may then break an application that inspects the error message string.

/tmp/error.js:1
throw new Error('oh no');
^
Error: oh no
    at Object.<anonymous> (/tmp/foo.js:1:7)
    ... TRUNCATED ...
    at internal/main/run_main_module.js:17:47

This output has been truncated, but the stack trace suggests that the error was thrown at line 1, column 7 of a file located at /tmp/error.js.

The following is an example of how the handler might be used to log a final distress message:

const logger = require('./lib/logger.js');
process.on('uncaughtException', (error) => {
  logger.send("An uncaught exception has occured", error, () => {
    console.error(error);
    process.exit(1);
  });
});

Promise.reject(new Error('oh no'));

(async () => {
  throw new Error('oh no');
})();

(node:52298) UnhandledPromiseRejectionWarning: Error: oh no
    at Object.<anonymous> (/tmp/reject.js:1:16)
    ... TRUNCATED ...
    at internal/main/run_main_module.js:17:47
(node:52298) UnhandledPromiseRejectionWarning: Unhandled promise
  rejection. This error originated either by throwing inside of an
  async function without a catch block, or by rejecting a promise
  which was not handled with .catch().

Unlike uncaught exceptions, unhandled promise rejections do not cause the process to crash in Node.js v14. In Node.js v15 and above, this will cause the process to exit. This behavior can be enabled in v14 by running the Node.js binary with the --unhandled-rejections=strict flag.

process.on('unhandledRejection', (reason, promise) => {});

Much like with the uncaughtException event, it’s important to not allow the process to continue running since it is likely in an invalid state. Consider running your Node.js processes with the flag enabled today to help future-proof your application. If you do encounter these uncaught rejection warnings while running an application in development, you should definitely track them down and fix them to prevent production bugs.

Node.js and the npm package ecosystem are both going through a transitional phase. Node.js was built with the callback pattern in mind for asynchronous activities, having the first argument of the callback be an error. It’s now adapting the promise/async function pattern. Applications you build today will have to deal with both patterns.

When an EventEmitter instance throws such an error, the following message will be displayed in the console before the process exits:

events.js:306
    throw err; // Unhandled 'error' event
    ^
Error [ERR_UNHANDLED_ERROR]: Unhandled error. (undefined)
    at EventEmitter.emit (events.js:304:17)
    at Object.<anonymous> (/tmp/foo.js:1:40)
    ... TRUNCATED ...
    at internal/main/run_main_module.js:17:47 {
  code: 'ERR_UNHANDLED_ERROR',
  context: undefined
}

Signals

$ kill -s SIGHUP <PROCESS_ID>

$ node -e "process.kill(<PROCESS_ID>, 'SIGHUP')"

Again, you should see the SIGHUP message printed in the console of the first application you’re running.

Now that you’re done experimenting with signals, you’re ready to terminate the original process. Run the following command in your second terminal window:

$ kill -9 <PROCESS_ID>

This command will send the SIGKILL signal to your process, terminating it immediately. The -9 argument tells the kill command to use the numeric version of the signal. SIGKILL is universally the ninth signal, so this command should be fairly portable and will work pretty much everywhere. Recall that the SIGKILL command can’t have a signal handler installed for it. In fact, if you were to attempt to listen for that event on the process event emitter, the following error would be thrown:

Error: uv_signal_start EINVAL

Avoiding Memory Leaks

const accounts = new Map();

module.exports.set = (account_id, account) => {
  accounts.set(account_id, account);
};

Why might an application be built this way? Well, it’s extremely fast. Writing a data change to an in-memory data structure will always be orders of magnitude faster than writing to an external service. It’s also very easy to make mutable globals like this in Node.js.

Bounded In-Process Caches

Caches should only be used in situations where performance requirements can’t be attained without them. Caches add an additional layer of complexity to an application. A cache also introduces the situation where the copy of the data in the cache may be outdated from the source of truth. For example, the account:123 resource may have been modified to have a balance of 0, even though the cached version still contains a balance of 100.

While cache invalidation philosophy is something specific to each organization, the requirement to avoid memory leaks is more universal. It’s safe to assume that a cache should never grow so much that it causes a process to crash.

Applications run in environments where there is a finite amount of memory available. A host machine will always have a maximum amount of physical RAM that it has available. Containers and virtual machines then have a smaller piece of that memory available. When a Node.js process consumes too much memory, it will either fail to get access to more memory, or a supervising process like Docker may terminate the process once a threshold has been reached. Memory is measured in the number of bytes being consumed, not the number of records being cached, so it’s good to use a tool that limits in-process cache size based on some semblance of the byte requirements of the data.

Next, substitute your username in one of those URLs and make another request. Then, use some other entries like express and fastify. Finally, circle back to the original tlhunter account again. This time, you should see that the request resulted in another cache miss. This is because lru-cache evicted the original tlhunter entry from the cache since newer entries replaced it and the cache had become full.

Another shortcoming is that the cache is only used by a single service instance! If you had a fleet of 100 web-api instances, each would still need to send a request for the same recipe-api resource at least once every 10 minutes. Each service also contains redundant caches, wasting overall available memory. This issue can be seen by running a second instance of the server and making a request to that:

$ PORT=4000 node server.js
$ time curl http://localhost:4000/account/tlhunter

Introducing Memcached

One of the most established caching services available is Memcached. It’s a dependable, no-frills cache, one that can be distributed across multiple machines. When instantiating a Memcached instance, you specify the maximum amount of memory that the instance may consume, and Memcached automatically purges newly added entries following the same LRU approach covered in the previous section.

Keys can be up to 250 bytes long, and values can be up to 1MB. Each individual key can have its own expiration time set.

There are two commands for performing string manipulations on the values stored in Memcached. The first is append(key, val, expire), and the second is prepend(key, val, expire). These commands allow an application to append and prepend a string to an existing value.

There are also two additional commands for making atomic changes where one client wants to ensure that another client hasn’t changed entries without it knowing. The first is gets(key), which returns both the value of the data and a “CAS” (Compare and Set) id. This is an integer that changes with each manipulation to the key. This value can then be used with a correlating cas(key, val, cas_id, expire) command. That command will set a key to a new value but only if the existing value has the same CAS id.

Various other commands exist for getting statistical information about the server, for retrieving the server settings, and for otherwise debugging the cache, though your applications probably won’t need to use them.

Running Memcached

Like many other Docker images, Memcached also includes an Alpine variant to consume less resources. When instantiating the Memcached service, there are a few flags that can be passed in, including -d to daemonize (not required with Docker containers), -m to set the maximum amount of memory (very useful), and -v to enable logging (this flag can be repeated to increase verbosity).

Run the following command in a terminal window to run Memcached:

$ docker run \
  --name distnode-memcached \
  -p 11211:11211 \
  -it --rm memcached:1.6-alpine \
  memcached -m 64 -vv

This Memcached instance is limited to 64MB of memory and will output a bunch of debugging information in your terminal. Port 11211 is the default Memcached port. Since the Docker command has the -it and --rm flags, you’ll be able to kill it with Ctrl + C when you’re done and the container will be removed from your system.

When running multiple Memcached instances, the instances themselves aren’t aware of each other. Instead, clients connect directly to the different instances and use a client-side hashing algorithm to determine which server contains a particular key. Ideally, this means each client uses the same server for the same key names, but it is possible for different client libraries to decide on different servers to store particular keys, which can result in cache misses and data redundancy.

Caching Data with Memcached

Now that you have your new service ready, execute two copies of the service in two separate terminals. In the first terminal, use the default port of 3000, and in the second terminal, override the port to be 4000, like so:

$ node caching/server-ext.js
$ PORT=4000 node caching/server-ext.js

Next, make a request to both of the services again. Hit the first service twice, and then hit the second service:

$ time curl http://localhost:3000/account/tlhunter # miss
$ time curl http://localhost:3000/account/tlhunter # hit
$ time curl http://localhost:4000/account/tlhunter # hit

In this example, the first request results in a cache miss. The service makes the outbound request to GitHub and then fills the cache and returns. In my case, this takes about 300ms. Next, the second request to the first service will result in a cache hit. The operation takes about 30ms in my case, which is a little slower than when I had run the process with just an in-memory LRU cache. Finally, the third request to the second service will also result in a cache hit, even though that service hasn’t made a request to GitHub. This is because both of the services use the same shared Memcached cache entry.

Data Structure Mutations

{
  "account": {
    "id": 7,
    "balance": 100
  }
}

Perhaps this representation of the cached entry is used by several different versions/releases of an application. Let’s refer to those as r1..r5. However, for the r6 release of the application, an engineer decides to change the shape of the cached object to be more efficient and to deal with an anticipated migration of account IDs from numbers to strings.

Running PostgreSQL

$ docker run \
  --name distnode-postgres \
  -it --rm \
  -p 5432:5432 \
  -e POSTGRES_PASSWORD=hunter2 \
  -e POSTGRES_USER=user \
  -e POSTGRES_DB=dbconn \
  postgres:12.3

Automatic Reconnection

Theoretically, if a database connection were to fail, then your application could terminate itself. Assuming you have infrastructure set up to detect such a termination, for example, a health check endpoint, then your Node.js process could be automatically restarted. That said, such infrastructure isn’t always available to an organization. Another thing to consider is that the overall application health isn’t necessarily any better by doing this. For example, if a process terminates and takes 10 seconds to fail a health check, then those are 10 seconds’ worth of failed requests. If an application loses connection to the database but is able to reconnect, that represents a potentially shorter period of downtime. For these reasons, developers often choose to implement reconnection logic.

Connection Pooling

When configured to use connection pools, an application will typically try to maintain a certain number of connections. When a connection goes down, the application attempts to create a new connection to compensate. When the application chooses to run a database query, it will then pick one of the available connections in the pool to pass the query through.

One reason to minimize database connections is that there is a finite number of connections that a database will accept. In fact, the default number of connections that a Postgres database will accept is 100. This number can be configured per database server. Managed Postgres installations like AWS RDS have different connection limitations based on tier.

If you go over the number of available connections, then the Postgres database server will refuse subsequent connections. This is something that you can simulate locally. The Postgres server that you’re running in Docker should be configured to have a maximum of 100 connections. Run the following commands in two separate terminal windows. The first will run the dbconn/pool.js service using up to 100 connections, and the second will hit the service with so many requests that it’ll be forced to use the entire connection pool:

$ MAX_CONN=100 node ./dbconn/pool.js
$ autocannon -c 200 http://localhost:3000/

Keep an eye on the terminal window where you’re running Postgres. While the tests run, you shouldn’t see anything bad happening.

$ MAX_CONN=101 node ./dbconn/pool.js
$ autocannon -c 200 http://localhost:3000/

This time, you should see the Postgres server complain with “FATAL: sorry, too many clients already” errors. Once the Autocannon test is complete, you should even see that the throughput is slightly lower.

If you would like to know how many connections a particular Postgres database is configured to handle (for example, when using a managed instance) run the following query:

SELECT * FROM pg_settings WHERE name = 'max_connections';

The maximum number of connections can be increased, but there is at least a small amount of overhead required for the server to handle the connections. If not, the default would be infinity. When choosing a connection count, you’ll probably need to make sure the number of connections used per process multiplied by the number of processes running at once is less than half of the number of connections the Postgres server can handle. This half part is important because if you deploy a new set of processes to replace the old processes, then there’s a small amount of time where both the new and old instances need to run with overlap.

So, if your server has a maximum of 100 connections available and you’re running 6 service instances, then the maximum number of connections each process can make is 8:

100 / 2 = 50 ; 50 / 6 = 8.3

One tactic I’ve seen at companies is that they’ll scale up beyond this maximum number of processes (like scaling up to 10 processes consuming a total of 80 connections). But when it’s time to do a deployment, they’ll scale back down the safe number of instances (6 in this case) during off periods, do a deployment, and then scale back up. While I can’t necessarily recommend this approach, I’d be lying if I said I wasn’t guilty of it myself.

Connection pooling isn’t just about resilience; it’s also about performance. The Postgres database, for example, isn’t able to handle multiple queries sent through the same connection at the same time. Instead, each query needs to finish before the following query can be sent, serially.

Configuring Knex

$ knex migrate:make create_users
$ ls migrations

The down() method performs the opposite operation. Technically, since the up() method performed two operations (creating a table and then adding users), the down() method should mirror those operations (deleting the users and destroying a table). But since dropping a table implicitly destroys the entries in it, the down() method only needs to drop the users table.

Next, run the following command to get a list of the migrations that Knex is currently aware of:

$ knex migrate:list
> No Completed Migration files Found.
> Found 1 Pending Migration file/files.
> 20200525141008_create_users.js

Applying a Migration

$ knex migrate:up
> Batch 1 ran the following migrations:
> 20200525141008_create_users.js

The knex migrate:up applies the next migration in line based on the order of migration filenames. In this case, there was only a single migration to be made.

$ docker exec \
  -it distnode-postgres \
  psql -U user -W dbconn

When prompted, enter the password hunter2 and press enter. Once that’s done, you’re now using an interactive Postgres terminal client. Commands entered in this client will take place in the dbconn database. For now, it would be useful to get a list of the tables stored in the database. Within the prompt, type \dt to do just that. When I run that command on my machine, I get the following results:

 Schema |         Name         | Type  | Owner
--------+----------------------+-------+-------
 public | knex_migrations      | table | user
 public | knex_migrations_lock | table | user
 public | users                | table | user

The users entry refers to the users table that was created when you ran the database migration. Next, to see the entries inside this table, type the command SELECT * FROM users; and press enter again. You should see results like these:

 id | username
----+----------
  1 | tlhunter
  2 | steve
  3 | bob

In this case, the three users that were created as part of the migration script are displayed.

The Knex query builder has converted the query that you represented by chaining JavaScript object methods into an equivalent SQL query. In this case, the table that is generated inside of the database could have been created by using the following SQL query:

CREATE TABLE users (
  id serial NOT NULL,
  username varchar(24) NOT NULL,
  CONSTRAINT users_pkey PRIMARY KEY (id),
  CONSTRAINT users_username_unique UNIQUE (username));

While you’re still running the Postgres client, it’s worth taking a look at the migrations table that Knex also created. Run another query, SELECT * FROM knex_migrations;, and press enter. On my machine, I get the following results back:

 id |              name              | batch |      migration_time
----+--------------------------------+-------+---------------------------
  2 | 20200525141008_create_users.js |     1 | 2020-05-25 22:17:19.15+00

In this case, the 20200525141008_create_users.js migration is the only migration that has been executed. Some additional meta information about the query is also stored. Since migration information is stored in a database, any developer would be able to run additional migrations for a remote database host, such as a production database, without the need to keep track of which migrations had been run previously.

The other table, knex_migrations_lock, isn’t as interesting. It’s used to create a lock so that multiple people don’t attempt to run migrations simultaneously, which could result in a corrupted database.

The only thing more exciting than one migration is two migrations, so go ahead and create another one. This second migration builds on the changes made in the first migration. Again, run a command to create a new migration file:

$ knex migrate:make create_groups

Rolling Back a Migration

$ knex migrate:down

In my case, when I run this locally, I get the following output:

Batch 2 rolled back the following migrations:
20200525172807_create_groups.js

Once this command has been run, the second migration will be rolled back, but the first migration will still be present. In this case, the SQL statements in the down() method of the create_groups migration have been executed. Feel free to run the knex migrate:list command at this point if you don’t believe me.

There’s no way that Knex can enforce that a down migration will completely undo the changes made by an up migration. That is ultimately up to the engineer to do. Unfortunately some operations just don’t have a correlating undo. As an example of this, imagine the following up and down migrations:

-- WARNING: DESTRUCTIVE MIGRATION!
-- MIGRATE UP
ALTER TABLE users DROP COLUMN username;
-- MIGRATE DOWN
ALTER TABLE users ADD COLUMN username VARCHAR(24) UNIQUE NOT NULL;

In this case, the up migration drops the username column, and the down migration adds the username column back. But the data that existed in the column has now been destroyed, and no amount of reverse migrations is going to get it back, either. What’s more, assuming there’s at least one user in the table, the down migration will fail because the unique constraint won’t be met—every username will be set to a null value!

One way these issues are sometimes discovered is after a code commit has been merged. For example, maybe a bad migration was merged and then run against the staging environment. At this point, all of the user accounts in the staging database have been corrupted and might need to be repaired. Some organizations copy production data into staging as part of a nightly task while anonymizing user data. In this case, the data will eventually be repaired in staging. Such safeguards aren’t always present for production.

In these situations, the migration should never be run in production. The way Knex works is that it runs each migration serially until the most recent is run. One way to fix these situations is to run the appropriate migrate down commands anywhere the database has been affected (in this case, the staging environment and the developer’s local environment). Next, delete the erroneous migration file entirely. This can probably be done in a single revert commit.

Live Migrations

Clients have many options to choose from for determining when a client is down. For example, they may choose to define a threshold before flipping the circuit breaker, such as if ten 500 errors are encountered within 60 seconds. Other approaches might involve checking the response time of a service, considering one to be down if it takes longer than 200 milliseconds to reply.

Things get more tricky when it comes to differentiating services from one another. For example, when you worked with Kubernetes, you created a Service object that abstracted the individual service instances away from you. In that situation it’s impossible to differentiate a faulty service instance from a healthy service instance. Luckily, Kubernetes may handle the health checks and can clean up a service automatically. With other technologies, such as Consul by HashiCorp, it’s possible to build a system where applications maintain an in-memory list of host and port combinations representing service instances. In that situation it’s possible to apply a circuit breaker on an individual-instance basis.

When it comes to communicating with outside services, such as the GitHub API, you’ll probably never know which underlying service instance is responding to a request; you just know that “the GitHub API is down.” In these situations, you may need to circuit-break the entire third-party API. This can be done by keeping a failure counter in a fast, in-memory database like Redis. When the number of 500 errors or ECONNREFUSED errors reaches a threshold, your services will then give up making requests and will instead fail immediately.

Exponential Backoff

const Redis = require('ioredis');
const DEFAULT = 5000;
const SCHEDULE = [100, 250, 500, 1000, 2500];
const redis = new Redis({
  retryStrategy: (times) => {
    return SCHEDULE[times] || DEFAULT;
  }
});

Depending on the operation, it may make sense to choose a different retry schedule. For example, for a service that depends on a database connection to function, this schedule might be fine. Once the application reaches the five second mark, it will continue to try to reconnect to the database forever. However, for other requests, like an HTTP request to an upstream service made as part of an incoming request from a downstream service, it wouldn’t be helpful to keep the incoming request open for too long. In that case, it might make sense to have a finite schedule containing three retries. When performance is more important, it also makes sense that the retries fire much quicker. For example, an HTTP retry schedule might look more like this:

10ms | 20ms | 40ms | quit

Random jitter can be introduced to the previous example like so:

const redis = new Redis({
  retryStrategy: (times) => {
    let time = SCHEDULE[times] || DEFAULT;
    return Math.random() * (time * 0.2) + time * 0.9; // ±10%
  }
});

The concept of jitter is useful in other situations as well. For example, an application may need to buffer stats in memory and flush it to a database every minute. This can be pulled off by making a setInterval(fn, 60_000) call once when the application starts. However, the same thundering herd problem exists. Applications are often deployed in a fleet all at once. This can mean that when deploying 10 applications, 10 flushes will happen at the same time every minute, periodically overwhelming the database.

Instead, jitter can be calculated randomly on a per-process basis when the process starts, where the jitter value is a number between zero and the interval of time. For example, when calculating an interval for an operation happening every minute, you might write code that looks like this:

const PERIOD = 60_000;
const OFFSET = Math.random() * PERIOD;
setTimeout(() => {
  setInterval(() => {
    syncStats();
  }, PERIOD);
}, OFFSET);

071 077 102 131 137 162 191 197 222

But without jitter, the request timeline would instead look like this:

060 060 060 120 120 120 180 180 180

Random Crashes