OpenResty Plus™ JSONB Library

This document covers the API of the lua-resty-jsonb library which replaces the lua-cjson library by avoiding serialization and deserialization altogether for both manipulating and storing the JSON data. This can greatly reduces the number of Lua GC objects created when dealing with JSON data while keeping the JSON semantics without migrating to a brand new serialization method.

This library provides an efficient binary JSON (called JSONB, but don’t confuse it with PostgreSQL or CockroachDB’s JSONB formats).

# Table of Contents

# Loading library

To load the library, we write:

local libjsonb = require "resty.jsonb"

# Compiling JSON

Compiling a JSON string to a JSONB binary string:

local jsonb, root_val_ref, err = libjsonb.compile(json)

The generated JSONB string can be used in place of the original JSON textual string for caching (like storing in lua_shared_dict, lua-resty-lrucache or even memcached or redis servers).

The 2nd returned value is a value reference for the root of the JSON value. Most of the API functions in this library manipulate JSON values by such special value references so that very few Lua GC objects are needed.

Almost all of the API functions in this library accept the JSONB string as the first argument. It also serves a kind of efficient C-level data structure. So no serialization or deserialization needed when either querying or storing the JSON data represented by the JSONB format.

The root_val_ref can also be accessed via libjsonb.root_val_ref in the case where there is no compile() calls in the current context.

# Serializing Lua Values to JSONB(TODO)

It is possible to serialize a Lua value (including a Lua table) directly to a JSONB string without going through JSON, as in

local jsonb, err = libjsonb.encode(lua_val)

# De-compiling to JSON

To generate JSON back from a JSONB string, we can do:

local json, err = libjsonb.decompile(jsonb)

Note that the generated JSON would not preserve the original object key order or any insignificant white-space characters. The user usually do not need to convert JSONB back to JSON for normal operations. It is only meant to be used for debugging.

Back to TOC

# Testing lua string

Testing if the lua string is a compiled JSONB string

local ok, err = libjsonb.is_jsonb(jsonb)

Back to TOC

# Fetching a sub-value by path

Get the value reference at json_data.foo[32].blah:

local val_ref, err = libjsonb.path(jsonb, root_val_ref, "foo", 32, "blah")

The foo and blah are object keys while 32 is an array index. The order here is important.

The value reference returned (as val_ref in this example) is a special value serving as a reference for the JSON value (which could be anything). To get the value reference of the JSON root, just pass no path component arguments to the path() function:

local val_ref, err = libjsonb.path(jsonb)

The 2nd argument does not need to be the root value reference. It can be any value references. For example, the previous example can also be rewritten as follows:

local foo_ref, err = libjsonb.path(jsonb, root_val_ref, "foo")
local elem_ref, err = libjsonb.path(jsonb, foo_ref, 32)
local blah_ref, err = libjsonb.path(jsonb, elem_ref, "blah")

Back to TOC

# De-referencing for Lua values

To de-reference a value reference returned by path() (or other functions) to a Lua value, do this:

local val, err = libjsonb.deref(jsonb, val_ref)

For performance reasons, the user should only de-reference primitive JSON values like numbers, strings, NULLs, and booleans. Otherwise new Lua tables would have to be created.

When a Lua number cannot represent a big JSON integer accurately, a LuaJIT FFI cdata object of the type uint64_t would be returned instead of a Lua number.

When the val_ref points to a JSON object or array, then the user can choose to feed a Lua table for the deref() function to use for the top-level JSON array or JSON object so that no new Lua table object needs to be created for the top-level JSON container value, as in

local tb = {}
tb, err = libjsonb.deref(jsonb, container_ref, tb)

Such input Lua tables can be recycled by libraries like OpenResty’s lua-tablepool.

Back to TOC

# Checking JSON value types

We can check the type of a JSONB value by checking its reference directly without de-referencing:

local typ, err = libjsonb.type(jsonb, val_ref)

The returned type string could be one of number, null, integer, array, object, boolean, and string.

Back to TOC

# Manipulating JSON arrays

We can manipulate JSON arrays without deserializing them to Lua tables.

Back to TOC

# Fetching the number of array elements

local n, err = libjsonb.nelems(jsonb, arr_ref)

Back to TOC

# Fetching an array element

We can use index() to fetch an array element by the array index, as in

local elem_ref, err = libjsonb.index(jsonb, arr_ref, 2)

This is equivalent to arr_val[2] (using 1-based indexes).

We use the path() function for it, just a little bit slower:

local elem_ref, err = libjsonb.path(jsonb, arr_ref, 2)

Back to TOC

# Iterating through the array

local n, err = libjsonb.nelems(jsonb, arr_ref)
for i = 1, n do
    local elem_ref, err = libjsonb.path(jsonb, arr_ref, i)
    -- ...
end

Alternatively, we could use the iterator style:

for elem_ref in libjsonb.elems(jsonb, arr_ref) do
    -- ...
end

Back to TOC

# Manipulating JSON objects

We can manipulate JSON arrays without deserializing them to Lua tables.

Back to TOC

# Fetching the number of key-value pairs

local n, err = libjsonb.npairs(jsonb, obj_ref)

Back to TOC

# Fetching a value for a key

local val_ref, err = libjsonb.index(jsonb, obj_ref, "foo")

We can also use the path function for it, just a little bit slower:

local val_ref, err = libjsonb.path(jsonb, obj_ref, "foo")

This returns a reference to the value for the key foo.

Back to TOC

# Iterating through the object

for key_ref, v_ref in libjsonb.pairs(jsonb, obj_ref) do
    -- ...
end

Back to TOC

# Manipulating JSON strings

It is also possible to manipulating JSON strings without de-referencing to Lua strings.

Back to TOC

# Fetching the length of a JSON string

local len, err = libjsonb.strlen(jsonb, str_ref)

Back to TOC

# Testing equality with a Lua string

local ok, err = libjsonb.eq(jsonb, str_ref, "some string value")

or discarding the letter-case:

local ok, err = libjsonb.eq_caseless(jsonb, str_ref, "ABCDEF")

Back to TOC

# Matching against a regex(TODO)

local ok, err = libjsonb.test(jsonb, str_ref, [=[^ \s* \w+ : \s* \d+ \s* $]=], "x")

Or fetching the sub-match captures too:

local match = require "table.new"(2, 0)
local ok, err = libjsonb.test(jsonb, str_ref, match, [=[^ \s* (\w+) : \s* (\d+) \s* $]=], "x")
local cap1 = match[1]
local cap2 = match[2]

Back to TOC

# Output to the response body stream

It is possible to output a sub-string in a JSON string value to the response body data stream without going through Lua strings:

local ok, err = libjsonb.print(jsonb, str_ref, offset, len)

It is functionally equivalent to the following code but without going through any Lua string objects:

local str = libjsonb.deref(jsonb, str_ref)
local ok, err = ngx.print(string.sub(str, offset, offset + len - 1)

Back to TOC

# API Aliases

It is convenient and also faster to use shorter API function names by introducing your own local Lua variables, like these:

local jb_eq = libjsonb.eq
local jb_path = libjsonb.path
local jb_elems = libjsonb.elems

Then we can use the shorter function names without typing crazy things like libjsonb.path everywhere.

Back to TOC

# TODO

The following features is still under development.

• LuaJIT FFI cdata object of the type uint64_t for big JSON integer.
• libjsonb.deref: dereference json object or array
• libjsonb.test: matching against a regex

Back to TOC

# Copyright & License

Copyright (C) 2020 by OpenResty Inc. All rights reserved.

This library is proprietary software and can only be used with a valid OpenResty XRay enterprise subscription.

Back to TOC