Benchmark, phase 1 · run p1-20261114 · Linux x86-64
pgrustj against PostgreSQL 18.6 + pgvector 0.8.7
On one x86-64 node, with each server alone in a matched 6-core, 24 GiB container and the same settings wherever both have the knob, pgrustj’s pgvector port gives the same recall as C PostgreSQL 18.6 + pgvector 0.8.7 at every search setting (within 0.005) and builds indexes of the same size. It is slower at almost every vector operation: 0.72–0.74× C’s HNSW search throughput at matched settings, and 1.90–2.13× C’s time for a serial HNSW build. As a plain PostgreSQL it is close to C (pgbench -S 1.10×, TPC-B-like 0.94×), and an hour-long mixed soak finished with zero failed transactions on both servers.
ef_search 100, 8 clients
0.72–0.74×
C’s queries per second on vector; 0.61× on halfvec
vector; IVFFlat 3.65–4.84×
pgbench -S
1.10×
C’s transactions per second, possibly client-bound
Every ratio on this page is pgrustj ÷ C. Above 1 is better for pgrustj in queries, rows and transactions per second, and worse in time, CPU and memory.
What was compared
- C
- PGDG PostgreSQL 18.6 with the PGDG pgvector 0.8.7 package, both written in C.
- pgrustj
- The roadius2/pgrust fork at
a9c5743a3c, built from source in its container (6 min 19 s,releaseprofile). It reports itself asPostgreSQL 18.6 (pgrust 0.4-beta). - Host
borg2: Intel i9-11900 (8 cores, 16 threads, AVX-512, F16C), 61 GiB, Ubuntu 25.04, Incus 6.0.6 on ZFS. The host also ran other instances.- Containers
- One database container at a time, six whole cores and 24 GiB each. The load generator ran in its own container on four other threads.
- Settings
- One file for both:
shared_buffers8GB,maintenance_work_mem8GB,work_mem64MB,max_parallel_workers_per_gather2,max_wal_size8GB. pgrustj also needsio_method=sync; C keeps its defaultworker. - Datasets
- From ann-benchmarks: glove-100-angular (1,183,514 × 100, cosine), as
vectorand ashalfvec; sift-128-euclidean (1,000,000 × 128, L2), asvector. Each has 10,000 held-out queries with their true top 100. - Indexes
- HNSW with
m = 16,ef_construction = 64; IVFFlat with 1184 lists on glove and 1000 on sift. Searches return the top 10 at 1, 8 and 16 clients; the client computes recall@10 from the ground truth.
Results
Search: recall against throughput
Each point is one search setting (ef_search for HNSW, probes for IVFFlat). Across: recall@10 from the 1-client pass over 10,000 queries. Up: queries per second at 8 clients. The curves sit at the same recall and separate in throughput: pgrustj’s points are the lower ones.
HNSW (one scale for all three)
IVFFlat (one scale for both)
Show the numbers
Index builds: wall time, median of 3
Serial builds and builds with 4 parallel workers, maintenance_work_mem 8GB. Shorter is better. Both sides’ indexes come out the same size on every row.
HNSW (seconds)
IVFFlat (seconds, its own scale)
Show the numbers
The one-hour soak: transactions per second, by 5-minute window
8 clients mixing HNSW searches (ef_search 100, weight 9) and inserts (weight 1), with a VACUUM every 10 minutes. Zero failed transactions on both servers; both stay flat. pgrustj served 0.79× C’s transactions over the hour.
Show the numbers
Benefits and negatives
Where pgrustj is equal or ahead
- Recall: the same at every search setting, within 0.0035 on HNSW and 0.0048 on IVFFlat, and in the exact scans.
- Index sizes: equal to the MiB on all 30 builds.
- Plain SQL:
pgbench -Sat 1.10× C, using 0.91× C’s CPU per transaction. Only 4.8 of 6 cores were busy on both sides, so the client may have been the limit.pgbench -itakes the same time. - Exact
vectorscan at 8 clients: 1.06–1.11× C’s throughput, with both servers using all six cores. Per CPU-second pgrustj looks ahead here, a lead confounded by C’s parallel-worker start-up and possible JIT. - Memory of a 4-worker HNSW build: 0.35–0.40× C’s container peak (3.45–4.19 against 9.98–10.60 GiB). C’s shared memory reserves the whole 8 GB graph area up front, most likely.
- Loading: 1.01× C on glove and 0.94× on sift.
- Stability: zero failed transactions in the hour, flat 5-minute windows, no memory growth after the first five minutes, and no error rows in 175 measurements a side.
- Portability: it builds from source and runs on Linux x86-64; the measured binary took 6 min 19 s to build.
Where pgrustj is behind
- Search throughput at matched settings, 8 clients: 0.72–0.74× on HNSW
vector, 0.61× on HNSWhalfvec, 0.76–0.84× on IVFFlat. p99 latency is 1.19–1.89× C’s. - HNSW builds: 1.90–2.13× C’s time serially and 1.58–1.66× with 4 workers on
vector; 2.74× and 2.19× onhalfvec. - IVFFlat builds, the largest gap: 3.65–4.84× C’s time serially and 2.75–2.97× with 4 workers.
- Exact scan at 1 client: 0.47–0.51× on
vectorand 0.24× onhalfvec. C runs it with two parallel workers; pgrustj’s planner builds no parallel plans. - CPU: 1.39–1.64× C’s per HNSW search query at
ef_search100, and 1.38× per TPC-B-like transaction. - Inserts under an HNSW index: 0.85–0.95× C’s rate. The DELETE + VACUUM after them takes 1.34–2.84× C’s time.
- Memory held: higher in every step except the 4-worker builds: 1.5–2.4× C’s container peak during searches and 1.74× at the soak’s peak.
- Soak throughput: 0.79× C’s; TPC-B-like 0.94×.
Why: causes and leads
A cause follows from the measurements. A lead is a likely explanation that nothing in this run measured on x86-64.
- cause The 1-client exact-scan gap on
vectoris parallelism. C’s container keeps 2.9 cores busy through a single-client exact scan, consistent with its planner’s 2 parallel workers; pgrustj’s keeps 1.00, because its planner, inherited from upstream v0.4-beta, builds no parallel plans. No plan was recorded, so the shape is inferred from the CPU. - cause Cells as low as 0.40× in the recall-target rows are a threshold effect. On sift, C’s 20 probes reach recall 0.9501 and pgrustj’s reach 0.9491, so pgrustj’s “first setting to reach 0.95” is 40 probes. At the same setting the ratio is 0.76×. The tables mark such cells †.
- lead Distance kernels. C pgvector’s x86-64 build vectorizes its distance sums (
-fassociative-math, an FMA clone, AVX/F16C/FMAhalfveckernels chosen at run time). pgrustj keeps C’s source-order scalar summation element by element, a deliberate fidelity decision. Every build gap is wider here than on the Mac’s arm64, and IVFFlat’s k-means, almost all distance work, has the widest. - lead
halfvechas no F16C route on x86-64. pgrustj converts half floats on a portable path here, against C’s hardware conversion. It is the weakest type: 0.61× search, 2.74× serial build, 0.24× exact scan at 1 client. - lead A fixed cost per search query. A line fit over the five
ef_searchsettings puts 0.5–0.7 ms more CPU per transaction on pgrustj’s side (1.84–2.06× C’s), plus 1.19–1.22× per unit ofef_searchonvectorand 1.50× onhalfvec. What the fixed part is spent on is not attributable from this data. - lead Hash-map inserts in the HNSW build. A Mac profile shows
HashMap::insertcosting about as much as the distance kernel. - lead Memory held. No cause attributed. Candidates to measure: 32 MiB of stack per connection thread, memory the engine keeps between statements, and how each side touches shared buffers.
Headline tables
The report’s three headline blocks, with the matched-setting rows moved to the top. A matched setting runs both engines at the same ef_search or probes, with each one’s 1-client recall in brackets; where the recalls agree, it is the like-for-like comparison. A “recall ≥ R” row takes the first setting whose recall reaches R, or the highest setting if none does (“not reached”).
† The cell’s setting, or the one before it, has a recall within 0.005 of the target. Where both engines’ curves agree that closely, which setting first clears the bar can turn on a noise-level recall difference and swing the ratio up to 2×. Read the matched-setting rows instead.
¹ Container peak is the database container’s memory high-water mark during the step, including whatever it already held; the file cache (ZFS’s ARC) is outside it. The build increment is rep 1’s peak minus the previous step’s peak, a lower bound on what the build added.
Exact scans, inserts and plain SQL
Exact scan, before any index
Inserts under the HNSW index, and the DELETE + VACUUM after them
Plain SQL, 8 clients × 60 s
Limits of this measurement
- One node, one run. Each search point is one 1-client pass and one 30 s
pgbenchrun per client count. Builds, loads and exact passes have three reps; inserts, plain SQL and the soak have one. Treat differences of a few percent as noise: build reps spread by up to 8.8%. - A shared host. Other instances ran on
borg2and nothing reserved the database cores. Before each step the harness waited for the host’s load to settle, but a neighbour starting mid-step went unseen. CPU frequency was not pinned. - Sequential, not interleaved. C ran from 23:36 to 05:05 UTC and pgrustj from 05:05 to 13:26 UTC, under different neighbour load.
- Warm cache only. Every search point follows a warm-up pass; nothing was measured cold.
- Different build origins. C is PGDG’s packaged build (GCC 13.3; whether it used
-march=nativeis not recorded). pgrustj was built from source for the default x86-64 target. - The baseline is PostgreSQL 18.6, the version pgrust ports, while PGDG also ships 18.7.
- Memory numbers are container high-water marks that include each step’s starting memory; the file cache is outside both containers. They are approximate in both directions.
- Recall ties count as misses. Recall is id-set overlap, so an exact distance tie at the 10th neighbour can count against either engine. Differences in the fourth decimal are noise. IVFFlat’s recall comes from one build per side.
- IVFFlat used 1184 lists on glove, one more than the design’s 1183, on both sides.
io_methoddiffers: pgrustj needssync, C runsworker. This touches every read and the WAL path.- No plan of the exact scan was recorded, and its parallelism is asymmetric. Checkpoints were not aligned with steps.
- Not equalised or not recorded: the OOM score adjustment, the cluster locale (C.UTF-8 against C), JIT (on for both, with different machinery) and data checksums.
- Client limits. The 1-client figures include the Python client’s own time per query, and the 1-client and 8-client points use different protocols, so their ratios are not like-for-like.
Reproduce and evidence
The scripts are in scripts/bench/; its README lists the site constants and prerequisites. The full report, with every table and its derivation, is docs/superpowers/reports/bench-borg-phase1-2026-11-15.md. The run used orchestrator a2e6b06f3b and pgrust a9c5743a3ca35502680d990125eed590b176d50a.
scripts/bench/sync.sh
ssh -n borg2 bash ~/bench/scripts/borg.sh create-db pgrust-db-c # once; also create-db pgrust-db-rust and create-client
ssh -n borg2 bash ~/bench/scripts/install-pg-c.sh
ssh -n borg2 'PGRUST_REF=a9c5743a3ca35502680d990125eed590b176d50a bash ~/bench/scripts/install-pgrust.sh'
ssh -n borg2 bash ~/bench/scripts/client/setup.sh
ssh -n borg2 bash ~/bench/scripts/client/datasets.sh glove-100-angular sift-128-euclidean
ssh -n borg2 'bash ~/bench/scripts/run-phase1.sh pg-c RUN_ID' # 5.5 h
ssh -n borg2 'bash ~/bench/scripts/run-phase1.sh pgrust RUN_ID' # 8.3 h, never overlapping
scp -r borg2:bench/results/RUN_ID ~/.cache/pgrust/bench-borg/
python3 scripts/bench/report.py ~/.cache/pgrust/bench-borg/RUN_ID