# Benchmarks

[`bench/compare.exs`](https://github.com/netoum/licet/blob/main/bench/compare.exs) runs Licet and SpiceDB side by side on one machine, against one Postgres server, with the same schema, relationships, and requests. This page describes the method, how to run it, and one set of results. The script lives in the GitHub repository and isn't part of the Hex package.

## Method

Both systems store their data in the same Postgres server, each in its own database: Licet in `licet_bench`, SpiceDB in `spicedb_bench`. Neither shares tables or locks with the other. Licet runs inside the benchmark process. SpiceDB runs as a server on the same machine and is called through its HTTP API.

The dataset is generated by the script and loaded into both systems before anything is timed. It has 25,161 relationships:

* 1,000 users and 100 groups of 20 users, where each of the first 50 groups also contains one of the others;
* a three-level folder tree of 10 root folders, 100 middle folders, and 1,000 leaves, where each root folder is viewable by one group;
* 10,000 documents in the leaves, each with an owner, and every fifth with a direct viewer.

```text
definition user {}

definition group {
  relation member: user | group#member
}

definition folder {
  relation parent: folder
  relation viewer: user | group#member
  permission view = viewer + parent->view
}

definition document {
  relation parent: folder
  relation owner: user
  relation viewer: user | group#member
  permission view = owner + viewer + parent->view
}
```

Each scenario sends the same request to both systems:

| Scenario | Request |
| --- | --- |
| Direct check | A user views a document they own. |
| Nested check | A user views a document through three folders and a nested group. |
| Denied check | A user who can't view the document, so the whole graph is walked. |
| `check_bulk` | 100 documents for one user, half allowed. |
| `lookup_resources` | The 109 documents, of 10,000, that one user can view. |

Before timing, the script requires both systems to give the same answer to every request.

Each scenario runs at two consistency levels:

* **Full:** Licet `consistency: :full` and SpiceDB `fullyConsistent`. Both read the latest data and neither uses its cache.
* **Default:** Licet's default `:minimize_latency` and SpiceDB's `minimizeLatency`. Both answer from a snapshot that is up to 5 seconds old, and both cache results for that snapshot.

After those two passes, the script grants and revokes 100,000 relationships on documents no scenario reads, so both systems keep that many deleted rows until garbage collection, and reruns the full-consistency scenarios.

Each one runs with one caller, then with eight concurrent callers. Benchee warms up for 2 seconds and measures for 5.

## Running it

Start Postgres, then SpiceDB with the Postgres datastore:

```shell
createdb -h 127.0.0.1 -p 54329 -U postgres spicedb_bench

docker run --rm --network host authzed/spicedb datastore migrate head \
  --datastore-engine postgres \
  --datastore-conn-uri "postgres://postgres:postgres@127.0.0.1:54329/spicedb_bench?sslmode=disable"

docker run --rm --network host authzed/spicedb serve \
  --datastore-engine postgres \
  --datastore-conn-uri "postgres://postgres:postgres@127.0.0.1:54329/spicedb_bench?sslmode=disable" \
  --http-enabled --grpc-preshared-key test
```

The [release binary](https://github.com/authzed/spicedb/releases) takes the same arguments. Then run the script from the project root:

```shell
MIX_ENV=test mix run bench/compare.exs
```

It creates `licet_bench`, loads the dataset into both systems, runs every scenario, and prints a summary table. `LICET_SPICEDB_URL` points it at another SpiceDB server; `PGHOST`, `PGPORT`, `PGUSER`, and `PGPASSWORD` at another Postgres. `BENCH_TIME` and `BENCH_WARMUP` change the seconds per measurement.

## Results

One run on an Intel i5-10300H (4 cores, 8 threads) with 8 GB of memory, Elixir 1.19.5 on Erlang/OTP 28, Postgres 15.19 in Docker, and SpiceDB 1.56.2 in Docker with host networking. Times are medians; `ips` is calls per second.

### Full consistency

| Scenario | Callers | Licet median | Licet ips | SpiceDB median | SpiceDB ips |
| --- | --- | --- | --- | --- | --- |
| Direct check | 1 | 1.46 ms | 608 | 1.52 ms | 617 |
| Direct check | 8 | 3.07 ms | 303 | 2.71 ms | 321 |
| Nested check | 1 | 6.17 ms | 155 | 1.48 ms | 638 |
| Nested check | 8 | 13.4 ms | 68 | 2.27 ms | 369 |
| Denied check | 1 | 4.97 ms | 186 | 1.31 ms | 699 |
| Denied check | 8 | 12.2 ms | 80 | 2.27 ms | 396 |
| `check_bulk`, 100 items | 1 | 13.9 ms | 66 | 7.11 ms | 128 |
| `check_bulk`, 100 items | 8 | 32.6 ms | 29 | 16.3 ms | 56 |
| `lookup_resources` | 1 | 21.4 ms | 43 | 6.55 ms | 142 |
| `lookup_resources` | 8 | 49.0 ms | 19 | 16.3 ms | 55 |

### Default consistency

| Scenario | Callers | Licet median | Licet ips | SpiceDB median | SpiceDB ips |
| --- | --- | --- | --- | --- | --- |
| Direct check | 1 | 20 μs | 39,600 | 0.75 ms | 1,200 |
| Direct check | 8 | 68 μs | 13,300 | 1.50 ms | 582 |
| Nested check | 1 | 15 μs | 46,000 | 0.75 ms | 1,200 |
| Nested check | 8 | 66 μs | 13,900 | 1.58 ms | 544 |
| Denied check | 1 | 23 μs | 36,200 | 0.76 ms | 1,200 |
| Denied check | 8 | 66 μs | 13,900 | 1.49 ms | 606 |
| `check_bulk`, 100 items | 1 | 1.23 ms | 745 | 5.62 ms | 170 |
| `check_bulk`, 100 items | 8 | 3.53 ms | 263 | 13.7 ms | 67 |
| `lookup_resources` | 1 | 20.2 ms | 48 | 5.49 ms | 168 |
| `lookup_resources` | 8 | 65.9 ms | 15 | 15.2 ms | 59 |

### Full consistency after write churn

After granting and revoking 100,000 relationships on unrelated documents:

| Scenario | Callers | Licet median | Licet ips | SpiceDB median | SpiceDB ips |
| --- | --- | --- | --- | --- | --- |
| Direct check | 1 | 1.48 ms | 621 | 1.87 ms | 467 |
| Direct check | 8 | 3.37 ms | 279 | 2.78 ms | 326 |
| Nested check | 1 | 6.62 ms | 145 | 1.89 ms | 455 |
| Nested check | 8 | 14.6 ms | 64 | 3.64 ms | 239 |
| Denied check | 1 | 7.03 ms | 131 | 1.94 ms | 466 |
| Denied check | 8 | 14.4 ms | 67 | 2.93 ms | 310 |
| `check_bulk`, 100 items | 1 | 30.3 ms | 32 | 11.1 ms | 88 |
| `check_bulk`, 100 items | 8 | 50.9 ms | 19 | 20.5 ms | 45 |
| `lookup_resources` | 1 | 28.0 ms | 35 | 7.27 ms | 131 |
| `lookup_resources` | 8 | 60.9 ms | 16 | 18.2 ms | 49 |

## Reading the results

With the default consistency, a repeated check is a lookup in an ETS table inside your application, so Licet answers in microseconds where SpiceDB needs a network round trip. This is the common case: most checks in an application repeat within the quantization window.

When nothing is cached, a check reads one level of the graph per query and covers every relation that level needs on the same objects. `check_bulk` groups checks that share a resource type, permission, and subject, so 100 items take about 14 ms instead of hundreds. A nested check still walks several levels in sequence and stays slower than SpiceDB; a direct check, which is one query on both sides, takes about the same time.

Deleted rows waiting for garbage collection are indexed by their lookup columns. After 100,000 such rows, single-check times stay close to the run without them. Bulk and lookup pick up some cost, as they do on SpiceDB.

`lookup_resources` walks back from the subject on both sides, and Licet doesn't cache lookups at either level. SpiceDB is about three to four times faster.

Keep in mind:

* SpiceDB pays for an HTTP round trip and its JSON gateway, while Licet runs in process. This is the real trade-off of embedding authorization, but SpiceDB's gRPC API would be somewhat faster than the HTTP gateway used here.
* SpiceDB runs as a single node. In production it usually runs several nodes that share their caches through dispatch, which this benchmark doesn't exercise.
* One laptop runs Postgres, SpiceDB, and the benchmark, so they compete for the same cores, especially with eight callers.

## Evaluator micro-benchmark

`bench/check.exs` times only Licet's evaluator: an in-memory check of a direct relation, with no Postgres.

```shell
mix run bench/check.exs
```

On the same machine, it runs at about 370,000 checks per second, with a median of 1.9 μs. Treat it as a baseline for the evaluator rather than for a running instance.