Click a column header to sort; click a row for the full per-gateway detail, sweeps included; tick up to three checkboxes to compare side by side. The Class chip is each project's own self-description, taken from its README or site, not our editorial. Rows are ordered by measurement, never by name. A muted cell means the lane was not measured; an explicit label means the suite probed it and recorded that the gateway does not serve it. Filters, sort, search, compare and the open drawer all live in the URL hash, so any view can be shared as a link.
Ingress protocol support matrix
Each gateway is probed across six ingress protocol shapes while the upstream mock stays fixed on the OpenAI shape, so every non-OpenAI cell is a translation claim: the gateway must translate the request out and the response back. One probe per cell, envelope validation rather than just the status code, and a passthrough guard that rejects the mock's own canned body as untranslated proxying.
Protocol matrix results are coming soon: no results/matrix JSONs are committed yet.
Green: served and envelope validated. Red: not served, with the probe status and body snippet as evidence. Grey: unprobed_auth, the gateway insists on an auth scheme (such as AWS SigV4) that this harness does not forge, recorded honestly instead of as a false red. Click a cell for the verdict note and body evidence.
Charts
Generated by the repo's chart script from the committed result JSONs. The highlight goes to whichever gateway measures best on each metric, in a neutral colour.
Method
Every gateway is measured the same way: one box, one instant Rust mock upstream that is never the bottleneck, one Go load generator, fixed CPU pinning (gateway on one core set, mock and loadgen on the others). Overhead is always gateway-minus-direct: the same load is run straight against the mock, and the direct baseline is subtracted, so the figure is what the gateway adds, not what the network or the mock costs. Hardware, architecture, build version and timestamp are stamped into every result JSON. A gateway is fully defined by its own directory: a one-file manifest declares how to build, launch and probe it, so adding or removing a directory adds or removes it everywhere.
perf/run.sh Latency and throughput
- Added latency (p50/p99, microseconds): at concurrency 1, gateway p99 minus direct-to-mock p99, small payloads.
- Max proxy RPS: highest sustained requests per second against an instant mock where p99 stays under 1000 ms and the error rate stays under 0.1 percent (CPU-bound ceiling).
- Sustained RPS at 20 ms: the same gates, but the mock sleeps 20 ms per request to model real LLM latency, so the ceiling is concurrent-in-flight capacity, the real production bottleneck. Same concurrency grid for every gateway.
- Idle RSS: process resident set after launch, before load.
- Peak RSS: maximum resident set while sustaining large-payload load at high concurrency.
- The mock answers stream:true with a paced SSE stream; the suite measures what the gateway adds on top of that pace.
- Added TTFT (p99): gateway first-content-frame time minus direct-to-mock TTFT at concurrency 1.
- Added per-token latency (p99): gateway content-frame gap minus direct-to-mock gap.
- Streams sustained: max concurrent streams where at least 99.9 percent of expected frames deliver, no stream stalls beyond twice the pacing interval, and the stream error rate stays under 0.1 percent.
- A gateway that answers 200 but buffers the whole response is recorded stream_served=false: measured, not hidden.
- The client speaks Anthropic (POST /v1/messages) while the upstream mock speaks OpenAI, so the gateway must translate both directions. The mock is untouched; that is the point.
- Added latency (p99) and sustained RPS at 20 ms on the translation path, same gates as perf. The direct baseline is the OpenAI shape straight to the mock, so the added-latency figure deliberately includes the translation work.
- A gateway that cannot serve Anthropic ingress against an OpenAI upstream records xlate_served=false with the probe status and body snippet as evidence.
- The same sustained-RPS-at-20ms sweep as perf, but with the gateway's native key/limit governance active: per-request virtual-key resolution, rate-limit accounting, and budget check-and-charge on the hot path.
- Each run also repeats the identical sweep against the plain, ungoverned launch, so one result file self-contains the governance overhead percentage from one box on one day, with no cross-run subtraction. Minted limits are generous enough that no cap ever trips: the cost measured is the check, not the limit.
- A capability suite, not a latency suite: one probe per ingress protocol cell (OpenAI, OpenAI Responses, Anthropic, Gemini, Cohere, Bedrock Converse) against the fixed OpenAI upstream.
- Envelope validation per cell, a passthrough guard that rejects the mock's canned bodies as untranslated proxying, and an unprobed_auth state for ingresses that demand signatures the harness does not forge.