Frontier · Gateways · On the Bench · AI tool benchmarks
How much throughput each gateway holds as you tighten the tail-latency budget: one concurrency sweep per cell, read at 1 ms, 5 ms, 10 ms, 50 ms, 100 ms, no bound.
Read across a row, not down a column: a flat row holds its rate even under a tight tail, and a row that climbs needs a loose tail to go fast.
| Gateway | Tested on | Req/s · 99% of requests under: | Rate held at its tightest bound | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 ms | 5 ms | 10 ms ▾ | 50 ms | 100 ms | no bound | ||||
| Busbar–busbar benchmark evidence | OpenAI | 62,231tail 795 µs | 62,587tail 3.3 ms | 62,587tail 3.3 ms | 62,587tail 3.3 ms | 62,587tail 3.3 ms | 62,587tail 3.3 ms | 99% of its full rate at 1 ms | |
| AISIX (api7)+1aisix benchmark evidence | OpenAI | 28,888tail 962 µs | 31,374tail 4.1 ms | 32,092tail 6.6 ms | 32,092tail 6.6 ms | 32,092tail 6.6 ms | 32,092tail 6.6 ms | 90% of its full rate at 1 ms | |
| APISIX+1apisix benchmark evidence | OpenAI | 12,187tail 845 µs | 23,089tail 3.3 ms | 25,474tail 5.7 ms | 26,590tail 19.9 ms | 26,590tail 19.9 ms | 26,590tail 19.9 ms | 46% of its full rate at 1 ms | |
| agentgateway+2agentgateway benchmark evidence | OpenAI | 21,156tail 588 µs | 22,239tail 4.6 ms | 22,380tail 9.4 ms | 22,545tail 22.7 ms | 22,545tail 22.7 ms | 22,545tail 22.7 ms | 94% of its full rate at 1 ms | |
| Kong+2kong benchmark evidence | OpenAI | 13,315tail 719 µs | 19,190tail 1.1 ms | 21,287tail 7.3 ms | 22,171tail 17.4 ms | 22,171tail 17.4 ms | 22,171tail 17.4 ms | 60% of its full rate at 1 ms | |
| Helicone+2helicone benchmark evidence | OpenAI | 15,405tail 750 µs | 16,288tail 3.1 ms | 16,376tail 6.3 ms | 16,376tail 6.3 ms | 16,376tail 6.3 ms | 16,376tail 6.3 ms | 94% of its full rate at 1 ms | |
| LiteLLM · Rust−5litellm-rust benchmark evidence | OpenAI | 8,404tail 588 µs | 10,049tail 2.5 ms | 10,049tail 2.5 ms | 10,049tail 2.5 ms | 10,049tail 2.5 ms | 10,049tail 2.5 ms | 84% of its full rate at 1 ms | |
| GoModel+2gomodel benchmark evidence | OpenAI | 4,555tail 897 µs | 6,685tail 4.0 ms | 6,862tail 8.8 ms | 7,415tail 33.8 ms | 7,475tail 66.8 ms | 7,475tail 66.8 ms | 61% of its full rate at 1 ms | |
| Higress−4higress benchmark evidence | OpenAI | 2,688tail 944 µs | 5,685tail 2.5 ms | 6,095tail 9.0 ms | 6,138tail 14.4 ms | 6,138tail 14.4 ms | 6,138tail 14.4 ms | 44% of its full rate at 1 ms | |
| Bifrost−1bifrost benchmark evidence | OpenAI | n/ano tested concurrency met this bound | 4,346tail 2.7 ms | 4,555tail 5.8 ms | 4,624tail 11.5 ms | 4,624tail 11.5 ms | 4,624tail 11.5 ms | 94% of its full rate at 5 ms | |
| Portkey–portkey benchmark evidence | OpenAI | n/ano tested concurrency met this bound | 831tail 3.7 ms | 835tail 5.0 ms | 836tail 14.7 ms | 836tail 14.7 ms | 836tail 14.7 ms | 99% of its full rate at 5 ms | |
| LiteLLM · Python–litellm-python benchmark evidence | OpenAI | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | 117tail 9.4 ms | 119tail 17.7 ms | 119tail 17.7 ms | 119tail 17.7 ms | 98% of its full rate at 10 ms | |
| One-APIone-api benchmark evidence | OpenAI | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | 32tail 34.2 ms | 32tail 34.2 ms | 38tail 2,530.7 ms | 84% of its full rate at 50 ms | |
| Planoplano benchmark evidence | OpenAI | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | 19tail 788.0 ms | served nothing under 100 ms | |
| TensorZerotensorzero benchmark evidence | OpenAI | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | n/ano tested concurrency met this bound | 6,275tail 44.0 ms | 11,879tail 53.1 ms | 12,324tail 110.0 ms | 51% of its full rate at 50 ms | |
Beside each name: places moved on the sorted column vs the August 2026 run (+ up, − down, – same place). Different engines measured the two runs; this is descriptive, not a statistical result.
Click a column to sort, a row for full detail. A muted cell carries no number, and its label and tooltip say why: not served, not declared, not measured, a rig fault, or (in red) a measured failure. Every table view, with its sort, bound, cell and filters, is shareable via its URL.
How to read this table
Each reading is the most requests/sec that cell carried while 99% of requests finished under that bound and it failed no request it accepted. Published as a single number, a flat row and a climbing row look comparable, and they are not the same machine.
The 10 ms column is the one the Performance tab ranks and is marked here; every other column is published on every cell too, so nothing is chosen for you. "≥" marks a reading whose sweep ran out of ladder while still qualifying: a floor, not a maximum. "tail" under a number is the tail that reading actually produced, never the bound.
A "0 · no tested concurrency met this bound" is a MEASUREMENT: the gateway served cleanly and no concurrency it was offered kept 99% of requests under that bound. It is not missing data - a cell with no measurement at all reads "no frontier" instead - and the difference matters most on the slowest rows, where five of the six columns can be that finding.
"N% of its full rate at B" in the last column is what the cell still carried at B, the tightest tail-latency bound it holds any rate at, as a share of its full rate with no latency bound at all. 99% at 1 ms is the good shape: the gateway gives up almost nothing even when you demand a 1 ms tail. A low percentage means it needs a loose tail to go fast. The bound matters as much as the percentage — 99% at 50 ms is not a flat gateway, it is a gateway that holds no rate at all under 50 ms — so the column sorts by that bound first and the percentage second, and a cell that held nothing under any published bound says so in words instead of showing 0%.
Per-cell throughput from the one 6x6 run; the cell chooser picks which cell every row reads.
What these numbers mean, from the engine's own constants
perf.frontierTHROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.