Strings store binary data and are the most basic data type available in Redis. In a sense, this is the only data type offered by Memcached, a competing cache service. If you strictly use Redis as a cache, then you might not ever need to touch another data type.

SET foo "bar"

When you type the SET command, the redis-cli REPL will offer hints as to the remaining arguments for the command. Many of the Redis commands offer more complex arguments, in particular when it comes to changing metadata. The full form of the SET command, according to the REPL, looks like this:

SET key value [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL]

GET foo
> "bar"

SET visits "100"
> OK
INCR visits
> (integer) 101

The first command sets a key named visits to the string value of 100. The next command increments the value of the key and returns the result; in this case, the result is the value 101. The INCR and INCRBY commands allow applications to atomically increment a value without having to first retrieve the value, increment it locally, and then set the value. This removes the race condition that was present in the single-threaded Node.js service you built in Example 9-1. Note that the return prompt displays some metadata about the result. In this case, it hints that the value is an integer. If you were to run the GET visits command, the value would be retrieved as a string again.

Note that if you hadn’t first set a value for the visits key, the INCR command would assume the missing value was zero. Redis assumes an appropriate empty value with most operations. This makes interacting with Redis in a distributed environment more convenient. For example, without this zero default, if you were to deploy a fleet of Node.js app instances, each of them incrementing the visits value when a request is received, you would need to manually set visits to zero before your applications run.

Redis has dozens of commands dedicated to operating on strings. Values can be appended to a string using the APPEND command. Bitwise read and write operations can be applied to a subset of a string, and increments can use floating point values using the INCRBYFLOAT command.

The list data structure stores a linked list of string values and is comparable to a JavaScript array. Much like a JavaScript array, entries are ordered and duplicates are fine.

RPUSH list aaa
> (integer) 1
RPUSH list bbb
> (integer) 2
LRANGE list 0 -1
> 1) "aaa"
> 2) "bbb"

Again, like with strings, Redis assumes the appropriate empty value for the list data type. In this case, when you ran the first RPUSH command, the key named list didn’t already exist. Redis assumed an empty list and added an entry to the list. The result of the RPUSH command is the length of the list, first returning a 1 and later returning a 2. Finally, the LRANGE command gets a list of entries in the list. Much like with JavaScript, Redis assumes list indexes are zero based. The first argument to LRANGE is the starting index, and the second argument is the end index. Negative values go from the end of the list, with -1 representing the final element, -2 the penultimate element, etc. The LRANGE key 0 -1 command can always be used to retrieve an entire list regardless of its length.

There are more than a dozen commands related to the list data type available in Redis. Table 9-1 lists many of the Redis list commands and their equivalent operation if performed on a JavaScript array.

Some of these commands may seem a little weird at first. For example, why does Redis need the RPOPLPUSH command when it could be rebuilt using a combination of other commands? It all comes down to the need to support many distributed clients performing atomic operations against data in a centralized location. If the RPOPLPUSH command didn’t exist, a client would need to perform both RPOP and LPUSH commands separately, which allows another client to interleave commands that can leave the data in an inconsistent state. “Seeking Atomicity” discuses such situations in more detail.

A Redis set is an unordered collection of unique values. It is comparable to new Set() in JavaScript. When inserting redundant values into either a JavaScript or Redis set, the redundant entry will silently be ignored.

SADD set alpha
> (integer) 1
SADD set beta
> (integer) 1
SADD set beta
> (integer) 0
SMEMBERS set
> 1) "beta"     2) "alpha"

The first SADD command adds an entry named alpha to a set named set. The second command adds an entry named beta to the same set. Both of these commands get a response of 1, meaning that a single entry was successfully added. The third SADD command attempts to add beta to the set again. This time, a 0 was returned, meaning no entries were added. Finally, the SMEMBERS command returns a list of each of the members in the set.

Table 9-2 is a list of some of the Redis set commands and their equivalent operations using a JavaScript Set.

Redis exposes several other commands for interacting with sets, notably commands for acting on unions and differences between sets. There is also the SRANDMEMBER and SPOP commands for reading a random entry of the set and for popping off an entry. The SSCAN command allows a client to iterate through the entries of a set while using a cursor, which is a way of performing pagination of results.

Similar to a list, a set that has all of its entries removed will result in its key being removed.

A Redis hash is a single key that contains multiple field/value pairs within it. A Redis hash most closely resembles a new Map() in JavaScript. Values within a hash are also treated as strings, though they do have some of the same operations available as normal Redis strings (like the ability to increment a value). Unlike normal Redis strings, the individual fields in a hash cannot have their own metadata applied (such as a TTL). When it comes to sharding, all fields in a hash will end up on the same machine.

HSET obj a 1
> (integer) 1
HSET obj b 2
> (integer) 1
HSET obj b 3
> (integer) 0
HGETALL obj
1) "a"      2) "1"      3) "b"      4) "3"

Table 9-3 is a list of some of the Redis hash commands and their equivalent operations using a JavaScript Map.

To increment a Map entry in JavaScript, you would need to first retrieve the entry, increment the value, and then set it again, assuming the map contains a value that is a Number instance. If the values contained an object with property v, then you could increment them with something like map.get(field).v++. The equivalent command using Redis is HINCRBY key field 1.

Consider that the string data type in Redis can hold anything that can be represented as a string of bytes. This includes a JSON object. With that in mind, why might you choose to use a hash instead of a JSON-encoded string? Hashes are useful when you want to store multiple properties close together, when all properties should have the same TTL, and when you need to atomically manipulate a subset of the keys. It’s also useful when the size of all the field values is so large that you wouldn’t want to retrieve the whole thing at once.

As an example of this, say that you have a 1MB JSON object representing an employee. One of the fields is the employee’s wages. The JSON representation for this might look something like this:

{"wage": 100000, "...other fields": "..."}

To modify the wage field in that document, you would need to call GET key to retrieve it, result = JSON.parse(response) to parse it, result.wage += 1000 to increment the wage, payload = JSON.stringify(result) to serialize it, and SET key payload to persist it. These modifications can’t easily be performed atomically because you’d need some sort of lock to prevent other clients from modifying the data simultaneously. There’s also overhead of reading and writing the 1MB payload, as well as for parsing and encoding the payload. By representing this data as a Redis hash, you’re free to directly modify exactly the field you want.

Since all the fields in a hash are stored together on a single Redis instance, it’s important to make sure that the majority of your data isn’t represented using a single massive hash. For example, if you wanted to store payroll information about every employee in Redis, it would be better to use a single key per employee instead of a single hash key with a field per employee.

A Redis sorted set is one of the more complicated data structures available in Redis. It stores a collection of unique string values that are sorted by numeric scores. Entries can be queried based on score ranges. JavaScript doesn’t have a built-in equivalent to a Redis sorted set, though one could be built using multiple data structures.

ZADD scores 1000 tlhunter
ZADD scores 500 zerker
ZADD scores 100 rupert
ZINCRBY scores 10 tlhunter
> "1010"
ZRANGE scores 0 -1 WITHSCORES
> 1) "rupert"     2) "100"
> 3) "zerker"     4) "900"
> 5) "tlhunter"   6) "1010"

The first three commands add entries to the sorted set. Calling multiple ZADD calls with the same member will replace the member’s score. The ZADD command returns a 1 when the member is new and a 0 when the entry already exists, much like with lists and sets. The ZINCRBY command increments the score of a member, assuming a score of 0 if the member doesn’t already exist.

The ZRANGE command retrieves a list of entries in the sorted set, based on score order. You can universally use the ZRANGE key 0 -1 command to get a list of all members in a sorted set. The WITHSCORES option instructs Redis to also include their scores.

Table 9-4 is a list of some of the commands available with sorted sets.

Using the leaderboard analogy, you can find out what the numeric rank of a player is by calling ZREVRANK scores tlhunter, which returns a 0 because it has the highest score. Many of the commands have a REV variant that treats the rankings in a reverse manner. Several also have a REM variant that removes the entry from the sorted set.

Most of the commands available in Redis are tied to keys with a specific data type. For example, the HDEL command deletes a field from a hash. But there are plenty of commands that either affect keys of any type or globally affect the Redis instance.

Table 9-5 contains some popular commands that affect a key of any data type.

Note that the KEYS command helps with local debugging but is inefficient and shouldn’t be used in production.

Table 9-6 lists some popular commands that interact with the Redis server in ways that aren’t associated with an individual key.

Note that the MONITOR command helps with local debugging but is inefficient and shouldn’t be used in production.

Other Types

Redis supports a few other data types and related commands that aren’t covered in this chapter.

One of these command sets deal with geolocation data. Internally, the geolocation commands operate on a sorted set containing entries scored by latitude and longitude values represented as a geohash. These values can be quickly retrieved using another command to find all the entries located within a configurable radius of a given latitude and longitude pair. This can be useful to do things like find all the businesses within a 1km radius.

There’s also a HyperLogLog data structure, which is a way of storing a compressed representation of a large set of data. This allows you to measure an approximate number of occurrences of an event. It’s useful for storing sampled data that doesn’t need to be 100% accurate.

Another interesting set of commands available in Redis is the PubSub (Publish/Subscribe) family of commands. These commands allow clients to subscribe to channels to receive messages or publish messages to channels. A copy of the message is sent to every client listening on the channel, though channels can have zero subscribers as well. This makes it convenient to blast information to several clients at once.

Now that you’re familiar with some of the theory behind Lua scripting, you’re ready to build something yourself. For this example, you’re going to build a waiting lobby for a multiplayer game. When players attempt to join a game, they are added to the lobby. If enough players have been added to the lobby, four players in this case, then the players are removed from the lobby and a game is created. A hash is created to contain a collection of actively running games and the players within them. At this point, the application could theoretically notify players that a game has started, but this is an exercise left to the reader.

For the first part of the application, you’ll create a Lua file containing the code to be executed on the Redis server. Create a new file named redis/add-user.lua and add the content from Example 9-4 to it. I bet you never thought you’d be writing Lua code in a Node.js book!

local LOBBY = KEYS[1] -- Set
local GAME = KEYS[2] -- Hash
local USER_ID = ARGV[1] -- String

redis.call('SADD', LOBBY, USER_ID)

if redis.call('SCARD', LOBBY) == 4 then
  local members = table.concat(redis.call('SMEMBERS', LOBBY), ",")
  redis.call('DEL', LOBBY) -- empty lobby
  local game_id = redis.sha1hex(members)
  redis.call('HSET', GAME, game_id, members)
  return {game_id, members}
end

return nil

The Lua scripting environment provided by Redis comes with two global arrays for accessing arguments provided to the script. The first is called KEYS, which contains the list of Redis keys, and the second is ARGV, which contains the normal arguments. The first key is assigned to a variable named LOBBY. This is a Redis set that contains a list of player identifiers. The local keyword is how Lua declares a local variable. The second key is assigned to the variable GAME, which is a hash containing active games. Finally, the only argument to the script is assigned to USER_ID, which is the ID of the player that was just added to the lobby.

Next, the player identifier is added to the LOBBY key. The Redis Lua environment provides the method redis.call() that allows Lua to call Redis commands. The first command being called in this file is the SADD (set add) command.

The next construct is where the first line of imperative programming happens (in this case, an if statement). This statement calls the SCARD (set cardinality) command to count the number of entries on the set. If the number of entries is not equal to 4 (which it isn’t for the very first run), then the if statement body is skipped. Then, the final line is called, and a nil value is returned. The nil value is then converted into a JavaScript null by the ioredis package.

However, once the fourth player has been added to the lobby, the if statement body will execute. The list of players is retrieved from the lobby by using the SMEMBERS (set members) command. This list of players is converted into a comma-separated string using the Lua table.concat() function. Next, the lobby is emptied. Recall that an empty list gets deleted, so in this case the DEL (delete) command is called to essentially clear the list.

Next, an identifier for the game is generated. There are many ways such an ID could have been generated, but in this case, a SHA1 hash of the members string is used. Lua doesn’t come with its own SHA1 function, but the Lua environment that Redis provides does. In this case, the function is provided via redis.sha1hex(). The string that is returned should be unique across all games, assuming the same players can’t join multiple games at the same time.⁴ This identifier is then set into the games hash using HSET, where the field name is the game ID and the value is a comma-separated list of player IDs.

Finally, an array (table) with two elements is returned, where the first is the game ID and the second is the list of players. Scripts can return data of different types between runs, and in this case, the script returns either a table or a nil.

This script atomically adds players to a lobby and creates games. It does require that both the lobby and the game hash be stored in the same Redis instance. You can ensure this happens either by using a single Redis instance or by using curly braces when naming keys. Normally, Redis chooses which instance to host a key on by hashing the key. However, if you wrap a subset of the key name in curly braces, only the value inside of the curly braces is used for the hash. In this case, if the lobby key was named lobby{pvp} and the game key was named game{pvp}, then the keys would always end up together.

The Lua script isn’t too interesting on its own, but things will get a little more exciting once you create a Node.js application.

This application connects to the Redis server, evaluates the script, and inserts four players. It’s rather basic and was built to illustrate how to call the commands, instead of integrating with a web server to expose a fully functioning game application.

Create a new file named redis/script.js and add the content from Example 9-5 to it.

#!/usr/bin/env node
// npm install ioredis@4.17
const redis = new (require('ioredis'))('localhost:6379');
redis.defineCommand("adduser", {
  numberOfKeys: 2,
  lua: require('fs').readFileSync(__dirname + '/add-user.lua')
});
const LOBBY = 'lobby', GAME = 'game';
(async () => {
  console.log(await redis.adduser(LOBBY, GAME, 'alice')); // null
  console.log(await redis.adduser(LOBBY, GAME, 'bob')); // null
  console.log(await redis.adduser(LOBBY, GAME, 'cindy')); // null
  const [gid, players] = await redis.adduser(LOBBY, GAME, 'tlhunter');
  console.log('GAME ID', gid, 'PLAYERS', players.split(','));
  redis.quit();
})();

This file begins by requiring the ioredis package and establishing a connection. Next, the content of the add-user.lua script is read and passed into the redis.defineCommand() method. This method abstracts away the Lua commands and has the application define a command using a chosen name. In this example, the script is aliased to a command named adduser.

Next, the two key names are declared that are used by the Redis Lua scripts. In this case, the lobby list key is lobby and the game hash is game. Theoretically, these key names can change on a per-call basis since they aren’t part of the scripts themselves. This could allow a game to have multiple lobbies, for example, one for silver-ranked players and one for gold-ranked players.

Next, the async function calls the redis.adduser() method four times to simulate four different players joining the lobby. The previous redis.defineCommand() method you called creates this new redis.adduser() method on the redis object. The arguments to this new method reflect the arguments passed to the Lua script (in this case, the lobby key, the game key, and the player ID). Note that this doesn’t create a command called ADDUSER on the Redis server; it’s just a local JavaScript method.

The calls to redis.adduser() will each run the add-user.lua script stored in Redis. The first three times it is called will each result in a null being returned. However, the final fourth call triggers the game creation logic. When that happens, an array is returned, with the first value being the game ID (gid) and the second returning the list of players (players).