Back from the lands of the midnight sun: Sweden, Finland and Norway. See the sky at midnight in June over Nordkapp overlooking the Barents Sea.

And what does the adventure have to show for it in the techno-blogging world? How about a Redis wire protocol implementation in Lua?

Redis Wire Protocol

The Redis database communicates with its clients using RESP. The RESP wire protocol runs over a bidirectional TCP connection—octets in, octets out.

The wire protocol itself is very simple—a line-based synchronous send-receive cycle where the connection client sends a command and then receives its reply. All Redis commands work that way, even if the reply is an “OK” status or error. Only one exception to this pattern occurs: Subscribing to a broadcast channel alters the protocol to a continuous receive cycle. Simple.

RESP Version 2

The protocol transports carriage-return line-feed terminated lines of text with one of the following prefixes to describe the format of the data item or items that follow within the connection channel.

Minus - for simple error strings
Plus + for simple non-error strings
Colon : for numbers, integers and floats
Dollar $ for bulk strings or nulls
Asterisk * for arrays

The resp module encodes in Lua as:

local _M = {
  null = {}
}

function _M.assert(...)
  if not select(1, ...) then
    error(select(2, ...), 0)
  end
  return ...
end

function _M.send(sock, data)
  return _M[assert("send"..type(data))](sock, data)
end

function _M.sendnumber(sock, data)
  return sock:send(":"..data.."\r\n")
end

function _M.sendstring(sock, data)
  return sock:send("$"..#data.."\r\n"..data.."\r\n")
end

function _M.sendtable(sock, data)
  local sent = _M.assert(sock:send("*"..#data.."\r\n"))
  for _, item in ipairs(data) do
    sent = sent + _M.assert(_M.send(sock, item))
  end
  return sent
end

local receive = {
  ["-"] = function(sock, rest)
    return false, rest
  end,
  ["+"] = function(sock, rest)
    return rest
  end,
  [":"] = function(sock, rest)
    return tonumber(rest)
  end,
  ["$"] = function(sock, rest)
    local len = tonumber(rest)
    if len == -1 then return _M.null end
    local data = _M.assert(sock:receive(len))
    _M.assert(sock:receive("*l") == "")
    return data
  end,
  ["*"] = function(sock, rest)
    local len = tonumber(rest)
    local data = {}
    for index = 1, len do
      local item = _M.assert(_M.receive(sock))
      data[index] = item
    end
    return data
  end
}

function _M.receive(sock)
  local data = _M.assert(sock:receive("*l"))
  return _M.assert(receive[data:sub(1, 1)])(sock, data:sub(2))
end

return _M

The resp.send function makes use of Lua’s type function to construct a function name for reflective dispatch. It looks up the function name within the module table and finally calls the function with the given socket and data. Hence calling resp.send(sock, {"LOLWUT"}) bounces to resp.sendtable. All the type-specific ‘send’ functions load in the module to allow clients to send explicit types if preferred.

Bulk Strings

The dollar prefix sends and receives bulk strings or nulls.

Note that a length of $-1$ is a Null Bulk String—not the same as an empty string. Clients should see a “null.” Lua uses nil for errors; nil as the first value and an error message as the second value for failure returns. Mapping the null bulk string to Lua’s nil would overlap the error space.

What to use as the ’null’ reply? Lua does not have a null value. Lua’s nil is not the same as null since table pairs with nil values do not exist; assigning nil to a table value removes the pair. It could use false or {}. Whatever value returned must not be nil and must be value equivalent, yielding true when compared using the == operator. An empty table {} meets this requirement provided that the resp module exports its identity so that callers can compare replies or elements of a reply with resp.null to determine if the reply equals ’null’ rather than an empty string. Comparing two tables using == compares them by their identity and not by their value. In other words, resp.null represents the ’null’ value.

Error Handling

The resp module provides its own version of assert. It exists in order to trigger errors without stack information about where the error occurs. It errors with a level of $0$ as the second argument meaning “send the error message only,” that is, without the stack traceback.

High-Level Redis

The RESP protocol handles the wire-level connection. For typical day-to-day use, however, a higher-level interface proves useful. Enter the hi module, providing the hi.redis function. It constructs a Redis connection which clients can call using a table or respond to send and receive methods.

local _M = {}
local resp = require "resp"
local socket = require "socket"
local socket_url = require "socket.url"

local metat = {
  __index = {
    send = function(redis, ...)
      return redis.try(resp.send(redis.sock,
        select("#", ...) == 1 and ... or { ... }))
    end,
    receive = function(redis)
      return redis.try(resp.receive(redis.sock))
    end
  },
  __call = socket.protect(function(redis, ...)
    redis:send(...)
    return redis:receive()
  end)
}

_M.redis = socket.protect(function(url)
    local parsed_url = assert(socket_url.parse(url or os.getenv("REDIS_URL")))
    assert(parsed_url.scheme == "redis")
    local sock = socket.tcp()
    local try = socket.newtry(function()
        sock:close()
      end)
    try(sock:connect(parsed_url.host, tonumber(parsed_url.port)))
    return setmetatable({sock = sock, try = try}, metat)
  end)

return _M

The implementation of hi makes use of socket.newtry for finalisation and socket.protect. Their use deserves some explanation.

The newtry function from socket module creates a function that acts like assert but with one additional useful feature: it invokes a finaliser function just before raising an error.

The Lua expression select("#", ...) == 1 and ... or { ... } also needs explaining. Sending data accepts either a single argument or multiple arguments. If a single argument, the sender passes it verbatim. If a single table, it sends the single table. However, if the caller passes multiple arguments, it wraps the Redis command within a table and ultimately sends it as an array of arguments. Naturally, Lua’s nil value does not pass. Wrapping nil values within a table effectively excludes the value because Lua tables cannot contain key-value pairs with nil values.

Examples

Worth explaining by a few instructive exemplars.

Laughing Out Loud What

See Wikipedia.

print(assert(require "hi".redis(){"LOLWUT", "VERSION", tostring(7)}))

Lua prints the following message when connected to the latest Dockerised Redis stack¹.

Redis ver. 7.1.241

Redis provides version-5 and version-6 LOLWUT messages.

The following subscribes to specific channels, x and y. Thereafter, the repeat loop will receive subscribe, unsubscribe and message triples. The example unpacks and prints each triple.

local hi = require "hi"
local redis = assert(hi.redis("redis://localhost:6379"))
assert(redis:send{"SUBSCRIBE", "x", "y"})
repeat
  print(unpack(assert(redis:receive())))
until false

Of course, it only succeeds if it finds a Redis server on localhost at the default port 6379.

Note that the loop never terminates. Redis receiving never times out. An adjustment will correct this. See below. This new version adds a one-second socket timeout and wraps the Redis receiver in a protected call. It removes the hi.redis argument allowing it to default to the environment variable setting for REDIS_URL.

local hi = require "hi"
local redis = assert(hi.redis())
redis.sock:settimeout(1)
assert(redis:send{"SUBSCRIBE", "ch:00", "ch:11"})
repeat
  local status, data = pcall(function()
      return redis:receive()
    end)
  if status then
    print(unpack(data))
  else
    if data == "closed" then break end
    assert(data == "timeout")
  end
until false

Note that a timeout does not trigger the finaliser and close the socket.

Set and Get JSON

The following exemplar presumes that the Lua path provides a JSON module capable of encoding and decoding JSON strings. It also assumes that the Redis server equips the JSON module, see modules.

local hi = require "hi"
local redis = assert(hi.redis())
local JSON = require "JSON"
assert(redis{"JSON.SET", "key", "$", JSON:encode{value = 123.456}})
local value = JSON:decode(assert(redis{"JSON.GET", "key", "value"}))
print(type(value), value)

Lua prints:

number  123.456

Notice that the JSON.SET and JSON.GET commands in Redis carry an extra ‘path’ argument specifying where to access data within the given key-addressed JSON object.

Conclusions

RESP version 2 is pretty straightforward. It almost seems trivially implemented and thereby suspiciously incomplete, though not the case. The protocol is complete for version 2; RESP has a third more sophisticated version—subject for another day.

Lua makes it easy thanks to its lightweight versatility: the programmer can do a lot with very little code. The perfect language does not exist, nevertheless. They all have their own list of strengths and weaknesses: their good, bad and ugly.

“When you have to shoot, shoot. Don’t talk.”—Tuco

Pull on redis/redis-stack:edge using Docker. Expose ports 6379 and 8001; the latter exposes a handy web interface. ↩︎

Redis Protocol in Lua