Benchmarks

bench/compare.exs runs Licet and SpiceDB side by side on one machine, against one Postgres server, with the same schema, relationships, and requests. This page describes the method, how to run it, and one set of results. The script lives in the GitHub repository and isn't part of the Hex package.

Method

Both systems store their data in the same Postgres server, each in its own database: Licet in licet_bench, SpiceDB in spicedb_bench. Neither shares tables or locks with the other. Licet runs inside the benchmark process. SpiceDB runs as a server on the same machine and is called through its HTTP API.

The dataset is generated by the script and loaded into both systems before anything is timed. It has 25,161 relationships:

• 1,000 users and 100 groups of 20 users, where each of the first 50 groups also contains one of the others;
• a three-level folder tree of 10 root folders, 100 middle folders, and 1,000 leaves, where each root folder is viewable by one group;
• 10,000 documents in the leaves, each with an owner, and every fifth with a direct viewer.

definition user {}

definition group {
  relation member: user | group#member
}

definition folder {
  relation parent: folder
  relation viewer: user | group#member
  permission view = viewer + parent->view
}

definition document {
  relation parent: folder
  relation owner: user
  relation viewer: user | group#member
  permission view = owner + viewer + parent->view
}

Each scenario sends the same request to both systems:

Before timing, the script requires both systems to give the same answer to every request.

Each scenario runs at two consistency levels:

• Full: Licet consistency: :full and SpiceDB fullyConsistent. Both read the latest data and neither uses its cache.
• Default: Licet's default :minimize_latency and SpiceDB's minimizeLatency. Both answer from a snapshot that is up to 5 seconds old, and both cache results for that snapshot.

After those two passes, the script grants and revokes 100,000 relationships on documents no scenario reads, so both systems keep that many deleted rows until garbage collection, and reruns the full-consistency scenarios.

Each one runs with one caller, then with eight concurrent callers. Benchee warms up for 2 seconds and measures for 5.

Running it

Start Postgres, then SpiceDB with the Postgres datastore:

createdb -h 127.0.0.1 -p 54329 -U postgres spicedb_bench

docker run --rm --network host authzed/spicedb datastore migrate head \
  --datastore-engine postgres \
  --datastore-conn-uri "postgres://postgres:postgres@127.0.0.1:54329/spicedb_bench?sslmode=disable"

docker run --rm --network host authzed/spicedb serve \
  --datastore-engine postgres \
  --datastore-conn-uri "postgres://postgres:postgres@127.0.0.1:54329/spicedb_bench?sslmode=disable" \
  --http-enabled --grpc-preshared-key test

The release binary takes the same arguments. Then run the script from the project root:

MIX_ENV=test mix run bench/compare.exs

It creates licet_bench, loads the dataset into both systems, runs every scenario, and prints a summary table. LICET_SPICEDB_URL points it at another SpiceDB server; PGHOST, PGPORT, PGUSER, and PGPASSWORD at another Postgres. BENCH_TIME and BENCH_WARMUP change the seconds per measurement.

Results

One run on an Intel i5-10300H (4 cores, 8 threads) with 8 GB of memory, Elixir 1.19.5 on Erlang/OTP 28, Postgres 15.19 in Docker, and SpiceDB 1.56.2 in Docker with host networking. Times are medians; ips is calls per second.

Full consistency

Default consistency

Full consistency after write churn

After granting and revoking 100,000 relationships on unrelated documents:

Reading the results

With the default consistency, a repeated check is a lookup in an ETS table inside your application, so Licet answers in microseconds where SpiceDB needs a network round trip. This is the common case: most checks in an application repeat within the quantization window.

When nothing is cached, a check reads one level of the graph per query and covers every relation that level needs on the same objects. check_bulk groups checks that share a resource type, permission, and subject, so 100 items take about 14 ms instead of hundreds. A nested check still walks several levels in sequence and stays slower than SpiceDB; a direct check, which is one query on both sides, takes about the same time.

Deleted rows waiting for garbage collection are indexed by their lookup columns. After 100,000 such rows, single-check times stay close to the run without them. Bulk and lookup pick up some cost, as they do on SpiceDB.

lookup_resources walks back from the subject on both sides, and Licet doesn't cache lookups at either level. SpiceDB is about three to four times faster.

Keep in mind:

• SpiceDB pays for an HTTP round trip and its JSON gateway, while Licet runs in process. This is the real trade-off of embedding authorization, but SpiceDB's gRPC API would be somewhat faster than the HTTP gateway used here.
• SpiceDB runs as a single node. In production it usually runs several nodes that share their caches through dispatch, which this benchmark doesn't exercise.
• One laptop runs Postgres, SpiceDB, and the benchmark, so they compete for the same cores, especially with eight callers.

Evaluator micro-benchmark

bench/check.exs times only Licet's evaluator: an in-memory check of a direct relation, with no Postgres.

mix run bench/check.exs

On the same machine, it runs at about 370,000 checks per second, with a median of 1.9 μs. Treat it as a baseline for the evaluator rather than for a running instance.