Benchmarks

What vci costs and what it saves, as measured. Every number says where and when it was taken, and links to the notes it comes from.

The chart on the overview is a model. This page has measurements only: production CI jobs of one large Rails suite, the plan step across fifteen releases, synthetic stores on a laptop, and a pilot against Turso's hosted service. Where two readings of the same thing disagree, both are here.

vex's Rails suite in production

ComposalAI/vex is a Rails application with 2,078 test files on 23 October 2026 (2,230 by the 25th), split over two CI jobs per pull request, with a seeded MySQL database and 16 test slots per job. Everything below is read from its GitHub Actions runs: the job logs, with vci's VCI_TIMING lines, and the jobs API for each step's start and end.

23 to 25 October: minutes per job

The first plain row is the median of 769 plain jobs in the week before vci took over, 13 to 19 October, as Measured on vex's Rails tests and CI is the bottleneck report it. The rows with vci are from the same pages, read from the logs of 23 to 25 October. A job where little changed was shorter than the plain job; a job where everything re-ran was longer, because each file runs in its own process so its inputs can be recorded, and about a thousand attestations are then signed, stored and pushed.

The plain figure did not hold up. Read again from the jobs API on 25 October (notes-rails-cost.md), the plain jobs of 16 and 17 October took a median of 272 s (application 261 s, services 284 s), and those of 18 October 304 and 318 s: 4.5 to 5.3 minutes, not 3.5. The 3.5 could not be reproduced. Since then the MySQL service image is pulled on every job, which takes container setup from 20 s to 48 s, plain or not.

25 October, vci 2.7.32: 143 jobs by runner kind

vex's runner pool hands out two kinds of host. The kind with 94,750 MiB of memory runs a test file's process in 4 to 5 s; the kind with 257,583 MiB takes 9 to 12 s, so every median here is per kind.

Against the plain 272 to 318 s, every case was longer on that day. The files that still ran after a small change were mostly long persistent_agents tests (90 to 280 s each, run as shards), so even the "little" case spent 186 s in its Ruby batch. A full re-run's 534 s on a fast runner was, in medians: the plan 23 s, the Ruby batch 406 s, checking, signing and storing 13 s, and the push 49 s. 57 of the 143 pushes went through a git merge-tree that failed after 8 to 55 s and then git-meta's own merge of 10 to 67 s; 2.7.34 and 2.7.35 fixed that (below).

26 October: the same files, plain and with vci, on the same runner

A manual workflow ran the application group (1,072 files, 6,463 tests) once per variant per job, with no store, no push and no shards:

On the same runner, vci's batch cost about what the plain run cost. The slowest file, unsharded, ran 284 to 319 s in one process and ended every vci batch, where the plain run spreads its tests over the workers. Loading the application before the slots fork made each process slower on the slow kind (7.9 s against 6.5 s); a compile cache and Process.warmup changed nothing measurable.

26 October: the new job, vci 2.7.35 and 2.7.36

At 00:30 UTC on 26 October vex moved its pull-request Rails jobs to file-backed SQLite, running 1,618 of the 2,232 files, on composal-x64-4x runners with 4 CPUs (vex #2128). At 05:46 UTC it moved them to composal-x64-8x, with 8 CPUs (#2139), the label of the Turso pilot below. So the three cases above cannot be measured again on the old job. These are the 32 pull-request jobs that finished between 04:46 and 17:45 UTC that day, with 4 or 8 slots. Most of those that finished before 06:00 ran on the 4-CPU label, and every later one on the 8-CPU label:

Three things outside vci dominate these medians. GitHub's cache restores took about 140 s each from about noon UTC, so the part of a job outside vci's steps was a median of 358 s on the 2.7.36 jobs against 72 s on the 2.7.35 ones. Two pull requests had 13 and 16 files failing for real, and each failing file runs again alone to tell a slot effect from a real failure. And 14 of the 24 jobs on 2.7.36 re-ran everything, because master had not yet been attested in that release's mode and the SQLite lane's attestations were new. For comparison, vex's own cold plain run of that lane took 12 min 28 s and 12 min 42 s per shard on 4 workers.

The push fix held: in those 32 jobs no git merge-tree failed, the seven jobs whose other job had pushed first merged in 0.5 to 2.4 s, and the publish step took a median of 22 to 27 s.

The plan step, release by release

The plan is what every job pays, whatever it runs. These are vex's Rails plans from 23 to 25 October 2026, 266 in all, read from the GitHub Actions logs.

Of the 15 seconds, over the last twelve pull-request plans of that period: 7.4 s fetching and merging the attestations master pushed since the job's cache was saved, 2.1 s downloading the attestation files the verdicts need, 1.6 s re-hashing inputs, 1.1 s for a second look at units that did not pass, and 0.7 s reading candidates. The fetch is git-meta's structural cost; a shared store answers a plan in about a dozen requests instead (below).

Two later readings, from notes-rails-cost.md: on 25 October, 2.7.32's plans took a median of 25 s (p90 46 s), 11.7 s of it fetching and merging; on 26 October, on the new SQLite job (4 CPUs until 05:46 UTC, 8 after), the action's plan step took a median of 18 to 22 s, of which vci's own plan total was 4.6 to 4.8 s and the rest installing vci and warming the store.

Checking, signing and storing a run

After its tests, vci run checks each unit, signs its attestation and stores it. 2.8.0 signs in process (ed25519 and ECDSA key files, and through ssh-agent) instead of starting ssh-keygen per unit, and writes a project's units in one transaction instead of a commit per write. Measured on a synthetic module of 1,200 Go packages, on a 14-core Mac busy with other work, medians of three runs, on 25 October 2026 (2.8.0 release notes):

The units are still checked one per CPU at a time: twice as many threads as cores grew the check's thread-seconds by half and did not shorten the step. On a heavily loaded machine, SQLite wrote a 1,200-unit run's 2,400 records in 4.8 s with a commit each, and in 0.9 s as one transaction.

The attestation stores side by side

The synthetic world: 1,200 Go packages, each attested at three commits, so 3,600 attestations of about 3 KB and 3,601 input manifests, written by three vci runs; macOS arm64, release build, two repetitions (STORE-BACKENDS.md, section 18, "Measured").

PostgreSQL, on a synthetic world of 600 Go units attested at three commits (1,800 attestations of about 3,100 bytes, 1,801 manifests of about 490 bytes), one connection, debug build, measured with a byte-counting proxy and pg_stat_statements (STORE-BACKENDS.md, section 17):

These manifests are tiny. On a real repository input manifests, often megabytes, dominate the bytes; none of this was measured on vex.

Request and statement counts, from the stores' tests:

• PostgreSQL: a plan of 1,200 units in 14 statements (the test allows 15), and 0 more for 1,200 manifest reads from 8 threads; a 1,000-unit batch written in 5 statements, BEGIN and COMMIT included.
• libSQL, release build against a local sqld: 1,200 units of 18 KB written in 0.64 s and 8 requests, then listed and read on a new handle in 0.10 s and 4 requests; synced into a new replica in 0.41 s and read from it in 11 ms. A plan's status and a prefetch of 300 units take at most 8 requests.
• SQLite: 1,200 units of 18 KB listed and read in about 8 ms, release build.

Turso's hosted service from a GitHub runner

On 27 October 2026 a pilot ran vci 2.8.0 against Turso's hosted service in aws-eu-west-1, about 25 ms of TCP round trip from vex's runners. It took three workflow runs (1, 2, 3), each on one composal-x64-8x runner, the label vex's Rails jobs use, with a fresh namespace per run. The project was vex-api's 2,291 Rails units. The git-meta side is vex's production path, from the 38 pull-request jobs of rails-tests.yml that ran between 00:12 and 01:50 UTC that day, mostly on vci 2.7.36.

Each Turso run wrote its units in four requests of 3 to 12 s: three of 8 MiB of values or more, and a smaller last one.

Turso gains the most in the plan: no cache to restore, nothing to fetch, and only the attestations a plan checks cross the network. Without the cache, on the pilot's own runners, git-meta's fetch and plan took 86 to 94 s. Storing into Turso takes longer than into git-meta's local store, but nothing is pushed after it: on the pilot's host class git-meta's store and push together took 43 to 81 s a job, against Turso's 22 to 42 s. On vex's smaller hosts (94,750 MiB) the same parts took 19 to 29 s on git-meta (3.9 to 6.6 s to store, 15 to 22 s to push); Turso was not measured there. A replica gained nothing on a fresh runner: synced from nothing, its plans took 4.1 to 5.1 s, and its first sync reads the whole namespace, about a millisecond a record, where a remote plan reads only what it checks. It suits a machine that keeps its clone. The pilot wrote 5,733 attestations, seven runs of 819, with none refused.

A vex pull request (#2214) proposes moving its Rails CI to Turso. It is under review.

Other measurements

• Measured on this repository: the demo projects, 21 units, with vci 1.5.0. A job with nothing changed costs about two seconds; attesting costs about 1.5 s per test file or package on top of its tests.
• Measured on vex-web: 253 node --test files, 17 s plain against 8 s with nothing changed, with vci 1.5.0.