Frontier · Gateways · On the Bench · AI tool benchmarks

How much throughput each gateway holds as you tighten the tail-latency budget: one concurrency sweep per cell, read at 1 ms, 5 ms, 10 ms, 50 ms, 100 ms, no bound.
Read across a row, not down a column: a flat row holds its rate even under a tight tail, and a row that climbs needs a loose tail to go fast.

15 of 15
Tail-latency bound
showing the req/s each gateway's chosen cell carried while 99% of requests finished under 10 ms
Cell shown
GatewayTested onReq/s · 99% of requests under:Rate held at its tightest bound
1 ms5 ms10 ms ▾50 ms100 msno bound
Busbar–busbar benchmark evidenceOpenAI62,231tail 795 µs62,587tail 3.3 ms62,587tail 3.3 ms62,587tail 3.3 ms62,587tail 3.3 ms62,587tail 3.3 ms1 ms: 62,231 req/s5 ms: 62,587 req/s10 ms: 62,587 req/s50 ms: 62,587 req/s100 ms: 62,587 req/sno bound: 62,587 req/s99% of its full rate at 1 ms
AISIX (api7)+1aisix benchmark evidenceOpenAI28,888tail 962 µs31,374tail 4.1 ms32,092tail 6.6 ms32,092tail 6.6 ms32,092tail 6.6 ms32,092tail 6.6 ms1 ms: 28,888 req/s5 ms: 31,374 req/s10 ms: 32,092 req/s50 ms: 32,092 req/s100 ms: 32,092 req/sno bound: 32,092 req/s90% of its full rate at 1 ms
APISIX+1apisix benchmark evidenceOpenAI12,187tail 845 µs23,089tail 3.3 ms25,474tail 5.7 ms26,590tail 19.9 ms26,590tail 19.9 ms26,590tail 19.9 ms1 ms: 12,187 req/s5 ms: 23,089 req/s10 ms: 25,474 req/s50 ms: 26,590 req/s100 ms: 26,590 req/sno bound: 26,590 req/s46% of its full rate at 1 ms
agentgateway+2agentgateway benchmark evidenceOpenAI21,156tail 588 µs22,239tail 4.6 ms22,380tail 9.4 ms22,545tail 22.7 ms22,545tail 22.7 ms22,545tail 22.7 ms1 ms: 21,156 req/s5 ms: 22,239 req/s10 ms: 22,380 req/s50 ms: 22,545 req/s100 ms: 22,545 req/sno bound: 22,545 req/s94% of its full rate at 1 ms
Kong+2kong benchmark evidenceOpenAI13,315tail 719 µs19,190tail 1.1 ms21,287tail 7.3 ms22,171tail 17.4 ms22,171tail 17.4 ms22,171tail 17.4 ms1 ms: 13,315 req/s5 ms: 19,190 req/s10 ms: 21,287 req/s50 ms: 22,171 req/s100 ms: 22,171 req/sno bound: 22,171 req/s60% of its full rate at 1 ms
Helicone+2helicone benchmark evidenceOpenAI15,405tail 750 µs16,288tail 3.1 ms16,376tail 6.3 ms16,376tail 6.3 ms16,376tail 6.3 ms16,376tail 6.3 ms1 ms: 15,405 req/s5 ms: 16,288 req/s10 ms: 16,376 req/s50 ms: 16,376 req/s100 ms: 16,376 req/sno bound: 16,376 req/s94% of its full rate at 1 ms
LiteLLM · Rust−5litellm-rust benchmark evidenceOpenAI8,404tail 588 µs10,049tail 2.5 ms10,049tail 2.5 ms10,049tail 2.5 ms10,049tail 2.5 ms10,049tail 2.5 ms1 ms: 8,404 req/s5 ms: 10,049 req/s10 ms: 10,049 req/s50 ms: 10,049 req/s100 ms: 10,049 req/sno bound: 10,049 req/s84% of its full rate at 1 ms
GoModel+2gomodel benchmark evidenceOpenAI4,555tail 897 µs6,685tail 4.0 ms6,862tail 8.8 ms7,415tail 33.8 ms7,475tail 66.8 ms7,475tail 66.8 ms1 ms: 4,555 req/s5 ms: 6,685 req/s10 ms: 6,862 req/s50 ms: 7,415 req/s100 ms: 7,475 req/sno bound: 7,475 req/s61% of its full rate at 1 ms
Higress−4higress benchmark evidenceOpenAI2,688tail 944 µs5,685tail 2.5 ms6,095tail 9.0 ms6,138tail 14.4 ms6,138tail 14.4 ms6,138tail 14.4 ms1 ms: 2,688 req/s5 ms: 5,685 req/s10 ms: 6,095 req/s50 ms: 6,138 req/s100 ms: 6,138 req/sno bound: 6,138 req/s44% of its full rate at 1 ms
Bifrost−1bifrost benchmark evidenceOpenAIn/ano tested concurrency met this bound4,346tail 2.7 ms4,555tail 5.8 ms4,624tail 11.5 ms4,624tail 11.5 ms4,624tail 11.5 ms1 ms: no tested concurrency met this bound5 ms: 4,346 req/s10 ms: 4,555 req/s50 ms: 4,624 req/s100 ms: 4,624 req/sno bound: 4,624 req/s94% of its full rate at 5 ms
Portkey–portkey benchmark evidenceOpenAIn/ano tested concurrency met this bound831tail 3.7 ms835tail 5.0 ms836tail 14.7 ms836tail 14.7 ms836tail 14.7 ms1 ms: no tested concurrency met this bound5 ms: 831 req/s10 ms: 835 req/s50 ms: 836 req/s100 ms: 836 req/sno bound: 836 req/s99% of its full rate at 5 ms
LiteLLM · Python–litellm-python benchmark evidenceOpenAIn/ano tested concurrency met this boundn/ano tested concurrency met this bound117tail 9.4 ms119tail 17.7 ms119tail 17.7 ms119tail 17.7 ms1 ms: no tested concurrency met this bound5 ms: no tested concurrency met this bound10 ms: 117 req/s50 ms: 119 req/s100 ms: 119 req/sno bound: 119 req/s98% of its full rate at 10 ms
One-APIone-api benchmark evidenceOpenAIn/ano tested concurrency met this boundn/ano tested concurrency met this boundn/ano tested concurrency met this bound32tail 34.2 ms32tail 34.2 ms38tail 2,530.7 ms1 ms: no tested concurrency met this bound5 ms: no tested concurrency met this bound10 ms: no tested concurrency met this bound50 ms: 32 req/s100 ms: 32 req/sno bound: 38 req/s84% of its full rate at 50 ms
Planoplano benchmark evidenceOpenAIn/ano tested concurrency met this boundn/ano tested concurrency met this boundn/ano tested concurrency met this boundn/ano tested concurrency met this boundn/ano tested concurrency met this bound19tail 788.0 ms1 ms: no tested concurrency met this bound5 ms: no tested concurrency met this bound10 ms: no tested concurrency met this bound50 ms: no tested concurrency met this bound100 ms: no tested concurrency met this boundno bound: 19 req/sserved nothing under 100 ms
TensorZerotensorzero benchmark evidenceOpenAIn/ano tested concurrency met this boundn/ano tested concurrency met this boundn/ano tested concurrency met this bound6,275tail 44.0 ms11,879tail 53.1 ms12,324tail 110.0 ms1 ms: no tested concurrency met this bound5 ms: no tested concurrency met this bound10 ms: no tested concurrency met this bound50 ms: 6,275 req/s100 ms: 11,879 req/sno bound: 12,324 req/s51% of its full rate at 50 ms

Beside each name: places moved on the sorted column vs the August 2026 run (+ up, − down, – same place). Different engines measured the two runs; this is descriptive, not a statistical result.

Click a column to sort, a row for full detail. A muted cell carries no number, and its label and tooltip say why: not served, not declared, not measured, a rig fault, or (in red) a measured failure. Every table view, with its sort, bound, cell and filters, is shareable via its URL.

How to read this table

Each reading is the most requests/sec that cell carried while 99% of requests finished under that bound and it failed no request it accepted. Published as a single number, a flat row and a climbing row look comparable, and they are not the same machine.

The 10 ms column is the one the Performance tab ranks and is marked here; every other column is published on every cell too, so nothing is chosen for you. "≥" marks a reading whose sweep ran out of ladder while still qualifying: a floor, not a maximum. "tail" under a number is the tail that reading actually produced, never the bound.

A "0 · no tested concurrency met this bound" is a MEASUREMENT: the gateway served cleanly and no concurrency it was offered kept 99% of requests under that bound. It is not missing data - a cell with no measurement at all reads "no frontier" instead - and the difference matters most on the slowest rows, where five of the six columns can be that finding.

"N% of its full rate at B" in the last column is what the cell still carried at B, the tightest tail-latency bound it holds any rate at, as a share of its full rate with no latency bound at all. 99% at 1 ms is the good shape: the gateway gives up almost nothing even when you demand a 1 ms tail. A low percentage means it needs a loose tail to go fast. The bound matters as much as the percentage — 99% at 50 ms is not a flat gateway, it is a gateway that holds no rate at all under 50 ms — so the column sorts by that bound first and the percentage second, and a cell that held nothing under any published bound says so in words instead of showing 0%.

Per-cell throughput from the one 6x6 run; the cell chooser picks which cell every row reads.

What these numbers mean, from the engine's own constants
Throughput at a tail-latency bound perf.frontier

THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.